A union of curiosity and data science

Knowledgebase and brain dump of a database engineer

Open non valid utf_8 character text file with python. (not unicode but non decodable bytes)


I attempted to read and write a CSV using python 3.5 and ran across the following error: 
Traceback (most recent call last):
File "C:\<File Path>.csv, line 14, in <module>
if __name__ == "__main__": main()
File "C:\<File Path>.csv", line 8, in main
lines = file.readlines()
File "C:\Programs\Python\Python35-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 4569: character maps to <undefined>

The original data appeared to have no character information and appeared as : Ã¢Â–¡â–¡â–¡â–¡

I added a couple of params to the open method and no more error : " encoding='ascii', errors='surrogateescape' "


def main():
    file =  open(r'\\filepath\filename.csv', 'r', encoding='ascii', errors='surrogateescape')    
    lines = file.readlines()
    for line in lines:
        print(line, end = '')

if __name__ == "__main__": main()


C# Dictionary from Key Value Pair String (URL)

I was searching for an elegant way to break a url into key value pairs and add them to a dictionary. 

Most of the code I came across appeared to be overkill so I wrote the function (getKeyValuePairs) below.

I return the dictionary from a function/method call because there seems to be some confusion in a few places on the web about being able to achieve this.

    class Program
        static void Main(string[] args)
            string message = "EventType=Information&EventDateGMT=2016-05-19 12:00:00&Source=Calling and Source Apps&Severity=99&Message=Hello World.!.";
            Dictionary<string, string> dict = getKeyValuePairs(message, '&', '=');
            foreach (var i in dict)  Console.WriteLine("Key: {0}, Value: {1}", i.Key, i.Value);

            Console.WriteLine("hit any key when ready."); Console.ReadKey();

        private static Dictionary<string, string> getKeyValuePairs(string data, char setDelimiter, char keyValueDelimiter)
            return data.Split(setDelimiter).ToDictionary(x => x.Split(keyValueDelimiter)[0], x => x.Split(keyValueDelimiter)[1]);


Preparing Visual Studio 2015 For Python Development

Setup a Visual Studio Python Environment in 5 Minutes using PTVS (Python Tools for Visual Studio)


You'll need:


Download the MSI:
Git "Python Tools for Visual Studio" : https://github.com/Microsoft/PTVS
2012 and 2012: https://github.com/Microsoft/PTVS/releases/v2.1.1
015 : https://ptvs.azureedge.net/download/PTVS%202.2.5%20VS%202015.msi


Install the tools:
Double click your MSI.


Create a new Visual Studio  Python Project:
File >New >Project : Installed >Templates >Python
Choose Python Application. 

PTVS Comes with many different python projects types including:

  • Django Web Projects
  • Flask /Jade Web Projects
  • Python Applications (Console Apps)
  • IronPython Projects


Setup the Python environment in VS (if not auto detected):

In most cases PTVS will find your python environment and set the location in VS. if this is not the case, follow the below steps to tell VS where your python environment lives.

Go to  Tools >Options >Python Tools >Environment Options. 
Click add environment, add a name for your environment. ex: "Python 3.4 Annaconda".
Add the path to your python.exe. 

Test your setup:

 In Visual Studio navigate to your PythonApplication1.py file and add the following line of code:
print("hello World")

Press F5 on your keyboard or click start. 

Congratulations! You're now setup for Python development using Visual Studio. Happy Dev'n.