pip install lxml – Permission denied

When using Python you mostly rely on pip to install necessary libraries. Recently I ran the following command on my Mac:

pip install lxml

The result was the following output:

error: could not create '/Library/Python/2.7/site-packages/lxml': Permission denied

Obviously, writing to the folder /Library/ is not possible for a normal user account. There are several possible options to overcome this problem.

  1. You can use the command sudo pip install lxml to install the package
  2. You can use the command STATIC_DEPS=true sudo pip install lxml to install the package. This will download libxml2 and libxslt and build the source distribution afterwards.
  3. You use virtualenv to install the package only locally in your current project directory. See here for a good tutorial of virtualenv.

From my perspective option 3 is the cleanest way to install and manage lxml.

XmlSerializer – Check if property was present in XML

Recently I was parsing a large amount of XML files using the XmlSerializer class. While implementing the parser I came across a requirement where I had to check if a given property was null because it was specified as null or because the value was not present in the XML.

At first this caused me some headache because accessing the underlying XML file violates the abstraction provided by the serializer. Luckily, I recognised that the generated entity class (generated from XSD.exe) contains a lot of properties with the name <PropertyName>Specified. This is a very neat feature of the XmlSerializer. If you add a property with that name scheme it automatically sets the property to true when the corresponding element is present in the XML.

The property could look like this:

[System.Xml.Serialization.XmlIgnoreAttribute]
public bool MyPropertySpecified { get; set; }

Note: XmlIgnore is necessary to avoid having the artificial property serialised into XML when using the XmlSerializer to write XML. More details on the behaviour can be found in the class documentation here.

Logstash – File input not working on Windows

Logstash is a great tool to transform the information stored in unstructured log files into a structured format. When using it on a Windows machine there are several things you should pay attention to (and which are not 100% documented).

Let’s say you want to use a file input and specify it in this way:

input {
  file{
    path => ["C:\Logs\*.logs"]
  }
}

When you run Logstash nothing happens and your files are not processed.

The reason for that is pretty simple: Logstash doesn’t like the \ and because of that it does not recognise the path properly. So simply change the config to look like this:

input {
  file{
    path => ["C:/Logs/*.logs"]
  }
}

Always use in Logstash configs and you will easily get around this problem. The problem is also known to the Logstash community (see this bug) but there is no fix in place yet.

Hint:

The mechanism for detecting which files have been written and which log entries are new is also not working correctly on Windows (see this bug here). The link also contains information on how to get around this problem.