Tuesday, 30 July 2013

Writing a home-monitoring system using python + internet...

Hmmm, what's that, you say? A Python home monitoring system? Why would anyone want to subject themselves to that kind of abuse?

Well, I , for one, amd lazy ... it shows in my work, it shows in my lifestyle, it shows in my weight .. and it shows in the way I write code. I'm the guy that will spend 2 days writing an app (which is available for purchase on the Google App Store) because I'm too lazy to get up and get myself a credit card ... because I'm too lazy to get up and scratch in the drawer for my old credit card so I can purchase stuff from Google Play ...

Before you go ranting that I expend time and effort to write an app which costs $3 ... you must remember that I am a programmer .. code it like water to me - extremely bitter, and mostly salty water, but water nonetheless ...

I eat code, I think code, I dream code and I can churn out code if I want to because it has become second nature to me. So it's simpler for me to just write it myself than it is to actually expend some physical effort to go and find something ... you cannot grep in the real world. You cannot hit [Ctrl][F] after opening your drawer. Frankly, I should patent some kind of "Reality Search" function which would apply itself to real-world objects ... but I'm too lazy to go down to the patent office to get it filed in person... Even if I could file patents online in my country, I would be too lazy to get up off my ass to go and use my new function...

Which brings me to the crux of my post ... why would I use python for such a serious task? What I am trying to do is definitely non-trivial, but surely PYTHON?

Well, the truth is that I am too lazy to go back to coding things in C++, I am not an Environmentalist, nor am I a packer ... I am more of a "mapper" who likes to interpret and adapt information into some kind of "view" which can be applied to knowledge ... and my "view" of the matter is that if I need the blazing speed to shoot myself in the foot a trillion times while simultaneously being bitten into millimeter sized pieces, then C++ is the way to go for every project.

If I were to use C++, then I would have to either write little command-line programs which did something specific with some command line input ... and tie that into my server (I'm using Apache - it's easy to set up, and doesn't require any movement on my part) ... or I could churn out some high-level code using a high-level "interpreted", "dynamically typed" language like Python... which saves on compile time, debug time, the amount of keystrokes I have to bang out ... it saves on heart rate as there is less agitation, and it saves on breathing since there is less cursing...

It's kind of a simple choice, really. For anything web-related, use web technologies ... for industrial "blazing speed", specialized applications, then get closer to the hardware with C or C++ ... For the record, I prefer the straightforward nature of C (I have spent several years coding industrial apps using C - if it's not straightforward, then you misunderstood something) but I love the speed you get when you can bang out a couple of object, an interface between them and then set them loose on the world of C++ ... so it's a trade off really. You just need to know which hammer will beat which nail, then all should be fine.

So back to the app - it's a very simplistic application:


  • Set up a "service" running in apache which allows an authenticated user to receive a screenshot containing all visible web cams on the home network.
  • Send these screenshots at some interval to the authenticated user.

This is fairly simplistic - since I will not even get into the hair raising mess of authenticating a user...
The simplicity is in how one would collate the images and send them over the network ... it's fairly straightforward to get the images - leave the DVR software running all the time, and periodically take a screenshot using this function:


import win32gui
import win32ui

def screenshot(_filename):
  wDC = win32gui.GetWindowDC(HWND_DESKTOP)  #hwnd
  _w = win32ui.GetSystemMetrics(SM_CXSCREEN)
  _h = win32ui.GetSystemMetrics(SM_CYSCREEN)
  dcObj=win32ui.CreateDCFromHandle(wDC)
  cDC=dcObj.CreateCompatibleDC()
  dataBitMap = win32ui.CreateBitmap()
  dataBitMap.CreateCompatibleBitmap(dcObj, _w, _h)
  cDC.SelectObject(dataBitMap)
  cDC.BitBlt((0,0),(_w, _h) , dcObj, (0,0), win32con.SRCCOPY)
  dataBitMap.SaveBitmapFile(cDC, _filename)

  cDC.DeleteDC()
  dcObj.ReleaseDC()
  win32gui.ReleaseDC(HWND_DESKTOP, wDC)

This would give me some image with a predetermined name which I could look up on the drive when I needed it ... I would probably store the images in a folder which represents the images from a specific cam on a specific day ...

Loading up the image is easy ... displaying it to the user is equally easy ... write a little JS function which requests an updated image via Ajax at some interval... Embed an <Object> into the HTML, tie it to the JS and VIOLA! You have a bandwidth-friendly solution to a home-surveillance problem...

This is obviously a gross oversimplification of the whole app, but it would definitely work along these lines ... I would obviously write some code (using PIX or Pillow) to convert the images to JPEGS on the fly either before saving, or at request time ... but the basis is the same. Obviously I would have to port forward on my router to allow that to happen ... but luckily DynDNS has made that much simpler, and all the grunt work of getting it set up , and obtaining a DOMAIN with a dynamic IP has been taken care of .. I have even assigned a static IP to my "surveillance" box which runs the apache server .. so most of the setup work is done .. all I need to do is to tie in some way of getting screenshots at a specified interval - which is equally easy ... write a "backdoor" into the system whereby I can leave a browser window open on the same machine running some periodic Ajax function which would force the server to respond at some predetermined interval - when I get a specific Ajax request, then do the hard work ... very hacky, but sufficient for my purposes - I could hack this together in a day. Now, imagine doing all of that, AND writing the C/C++ code to back it up? Can you imagine the pain and agony and suffering you would go through simply debugging why X won't initialize before Y?

If I wanted ultra-secure 128bit encryption and blazing fast real-time RTSP streaming web app, then I would be forced to resort to a C/C++ HTTP server which relies on the live555 media streaming library somewhere... and then write a C/C++/wxWidgets-backed client side which decodes and displays using FFMPEG... but, thankfully, I don't need to go that route just yet :-)

Thanks to pyfunc over at StackOverflow for the screenshot function, which I am generally going to plunder from the SO site - as much as I plunder many lines of code from awesome coders who are not as lazy as I am.





On working with Python and Javascript .. and HTML and Ajax and SQLite

I have recently been creating a production management system. The system is not that complex, but it is taking an inordinate of time to write ... even though I am using rapid-prototyping languages - Python, SQLite3, HTML4/5, Javascript coupled with liberal Ajax sprinkles ...

I simply don't understand - but I will try to get to the root cause.

So far, I have constructed the tables and written a database backend wrapper - for the SQLITE3 DB which I manipulate on the server side using python cgi.

I have created the concept of "sessions", but not in the MOD_PYTHON way, no, in the more traditional - hacky, store-it-in-the-window_name way ... By allowing the user to send an Ajaxlogin request, which generates a session token on the server side, which is then "kept" or "remembered" on the client side by storing the token in the window name browser object variable... It's probably the most rubbish way I can do it, but it works - survives page reloads (F5), survives navigation buttons (back, forward) ... and means I end up writing less python code to deal with IP-address-based-sessions since our network is mostly DHCP based, and we frequently restart our routers in any case due to some other technical difficulty we have to live with due to a third party.

I have added support for individual "users" and am in the process of linking "projects" to users in a nice way.

I have created an HTMLFormatter python script which can generate nicely formatted links which call JavaScript functions ...

I have written JavaScript functions which append the session token to outgoing requests ...

I have written more python cgi code which checks for session tokens and discovers which user "owns" the token  and does something fancy with that ...


I think I'm starting to notice a trend here ...

Medium/Large frameworks hide all this detail from the programmer - allowing them to deal with the core of their code - the business logic. When writing something from scratch, we tend to spend an inordinate amount of time getting a trivial feature to work "just right", only to blow another week on it when we discover it bombs out in IE...

When you are a hobbyist programmer, it makes sense to write as much as you can on your own so that you can both sharpen skill and learn a diverse set of  code "katas". But when you are an experienced programmer who is simply trying to get a project "done", you don't want to spend 2 days learning how to handle sessions which do not use cookies ... you want to spend that time banging out the code that ties the project to the user and allows the bosses to draw up time sheets and plot graphs and all those other things that bosses do with production management systems....

Even for hobbyist programmers I would recommend using frameworks for things you are already comfortable with - know how to implement sessions already? Use MOD_PYTHON for the next project which requires sessions - you know how it works, now use some code which saves you from 3 weeks of implementing it yourself.

Keep it simple ... but simple is a very relative term - upon embarking on this project, my idea of simple was to have a simple "session" - based on the user IP, which displayed their own project queue to each user, and a more "admin" interface to the sysadmin. I spent about a day banging out some nice python/html/JS/Ajax code to get it all working nicely and discovered that my idea fell flat when the DHCP server (our router) was rebooted during a session ... so I spent some more time tweaking the idea to use ip +  machine name ... but that quickly fell flat when I discovered that users on our network didn't have unique machine names ... and I wan't up to the task of renaming each machine to make my code work ... which is when I stumbled on the "shared login" concept - log a request to the server to get a session token, then associate the token to a user id on server side, and store only the session token on the client side - that way we don't care if the transport mechanism goes offline between requests - we simply send off each request with the session token and let the server handle the rest...

This was by far the most robust approach I have found so far - now if only I could get it to work if the user suddenly starts working in a different tab .. hmmm ...

Well, the only thing left to say is that python/JS/HTML+Ajax are "rapid prototyping" tools for a simple reason - somebody has already built the prototype, whether is is a python recipe or a JS library, it has been built - if you are going to keep inventing the wheel it will be slow in any language ... and will probably end up gathering many patches along the way, until it ends up like my project - which now consists of more patch than actual wheel ....