Placidity - Part 8, Application I

So far we have implemented various bits and pieces needed by our interactive interpreter application. Finally it's time to integrate those parts and make it possible for the user to give it a go. First I'm going to implement it using old skool methodology (code first). After that I will restructure the code so that it's easier to test and sort out possible issues of which there are quite a few.



First Attempt

Now that we have some components (PluginLoader, Interpreter, etc.) it's time to wire them together to form an Application. Here's my first attempt:

application.py:


I wrapped the logic needed by plugin directory in a class of its own. You can find the full implementation here. I also implemented a naive loop that polls user input, feeds it to the interpreter and outputs the result.

To run the application I set up a main program in placidity.py (this goes to /placidity/ just like application.py). It looks like this:

placidity.py:


Now it's possible to run the application just by invoking "python placidity.py" on the command prompt. Of course, as expected, not everything went smoothly. First of all it failed at File.__init_classes as I had forgotten to check if a path is a directory. I added a quick hack to fix the issue (the "XXX" part). This is a special case that should be mocked and tested properly.

After fooling around with the interpeter a bit I noticed that commands just didn't work properly. For instance "help" invoked the help command of Python even though it has been specified that it should invoke the native "help". Also functionality such as assignments didn't work out of the box. At least simple math worked!

Again this was due to an oversight in testing. As I happen to use Win7 as my main development platform and I expected that paths are provided in Unix format for my File construction, it fails there. So yet another thing to test for.

Let's test those special cases and see if we can get it work better.

Testing Special Cases

Although the fix for the File path issue found above made the interpreter run, it's not correct. It's clearest to fix the issue by mocking the case that triggered it properly and by determining the correct behavior that way. Now the question is how to do this.

In this case the logic is quite clear. A File should be able

1. construct parent graph based on the path
2. if the end of the path is a directory, it should check its children and construct graph appropriately (if a child is a folder, it should do the same check).
   

So far we have tackled part 1. adequately. Part 2. needs to be tested and implemented properly. I checked out Python's API and figured out that in order to handle 2. I need to patch and mock "os.listdir" in addition to "os.path.isdir" used in the hack before.

Here are the tests I came up with:

test_file.py:


There are a few things to notice. If you take a look at "test_python_files_in_folder" you can see that I patched "os.path.isdir" and "os.listdir" manually. I did this because their return value depends on input. Should I just use "return_value" attribute and regular patching, I would just end up in a recursion trap.

Besides this I enforced a separator rule to handle separator differences between Unix and Windows platforms. I ended up implementing a specific method, separators_test, that handles this aspect of testing. All you need to do is to provide a test function to it and it does the rest.

I also added a test for a case in which no path is provided to File and made sure it works properly.

Here's my current implementation:

file.py:


Sadly the fixes made above didn't make the interactive interpreter run properly yet. When I ran it, it gave me "AttributeError: 'NoneType' object has no attribute 'classes'" error. After looking at the error properly, I figured out that there must be something fishy going on at the File class. It's as if the file class introspection is totally broken in reality.

Fixing File

On retrospect it probably wasn't the greatest idea to load a Python module by first stashing it to a temporary file considering normally it already exists in the file system. Why not to just load it directly then? Clearly the part dealing with a temporary file belongs to the test instead of the implementation. I will change that next.

By inspecting the implementation of File, we can see that __init_classes gets called for each file. It does not make any difference between based on a file type. It might be a good idea to enforce a rule that makes it operate only on Python (.py) files. I will just add a simple test in the implementation for now. Later this case should be mocked properly, however.

You can find the fixed files here.

Separating the handling of a temporary file from the implementation to the test cleaned up tests considerably as it allowed me to get rid of mocking "with" altogether! The interactive interpreter still fails to run, though. And it still gives the same error! I will inspect the plugin loader next as that's where it really fails.

Fixing Plugin Loader

The tests designed for plugin loader don't match reality. There are two main problems:

1. The mocked files don't have any extension.
2. There are no compiled files (.pyc) amongst the mocked ones.
   

Fortunately it's easy enough to mock out those cases. You can find my fixed plugin loader here. As you can see, I had to define a new attribute, type, for File. Obviously I need to implement that too. Tests and implementation adding it may be found here.

The most interesting part of the tests is "test_find_by_name_and_type" method that verifies that the type has been set appropriately and that "find" method with it.

Now the interpreter should run without any errors. It's far from perfect though! For instance I noticed that if you happen to have some extra files in the commands folder, it fails. This is a special case that should be tested for. Also the interpreter does not handle matching properly yet. A good example of this is the "clean" command that causes "AttributeError: class Clean has no attribute 'matches'". This is definitely something that needs to be taken care of.

Fixing "matches" and "execute"

The above error is caused by the fact that a plugin may omit the implementation of a "matches" method. The idea is that in this case a default implementation (just direct string comparison) should be injected to the class just before its fed to the interpreter. It looks like PluginLoader should handle this. You can find my test and implementation for this case here.

Now running and trying out the "clean" command gives following error: "AttributeError: 'Commands' object has no attribute 'keys'". Checking out interpreter tests shows what's causing the problem. Apparently the execute methods used in testing have faulty signatures. They are missing "self"! You can find the fixed version of interpreter here.

Summary

Guess what? Our own little Frankenstein is alive! Here's a sample output:

a=4
None
b=13
None
a+b
17
variables
Stored variables:
a=4
b=13
vars
<built-in function="" vars="">
clean
None
variables
No stored variables
quit
Use quit() or Ctrl-Z plus Return to exit

The output is far from perfect. It outputs redundant "None" on assignment, it might be nice to have some sort of prefix (ie. ">>>" or similar), "vars" seems broken (conflict in the interpreter) and "help" is broken (conflict) at least. It's on the right track, however! I will look into fixing these issues in the next part of the series. You may find the source code of this part of the series here.