Overriding system libraries

Regression testing is great, but when you're testing your applications interactions with other libraries (especially ones that do lots of work) while doing memory checking (valgrind) things can get really really slow. Opendias testing is no exception.

Since we don't want to test these libraries - we just want to test opendias - I've decided to 'stub out' some of the larger application libraries (some parts of same, tesseract and PDF parsing) while running regression tests.

Some very cool stuff. Here's two links I've recently found very useful in that task.

• http://www.techytalk.info/c-cplusplus-library-programming-on-linux-part-two-dynamic-libraries/
• http://entrenchant.blogspot.co.uk/2009/03/linking-static-libraries-into-shared.html

I'm pleased to say, they do their job very well, and I'm very happy at the results.

Serious Authentication

The big news for the 0.9 series is the migration of location based privileges to a proper user authentication mechanism.
The first stage of the is making the application session aware, and that's what's been going on recently.
This looks to be going well and give us some extra flexibility (with fancy user specific functionality in the future.)
Once things are stable on the session handling, user profiles/ management/ authentication and finally migration of location based privileges, will happen.

Let's just get on with it!

So, I've created a new development branch for the 0.9 series.

First on the list is removing all the hard coded directory locations. Hopefully by the end we'll be able to install the application into any location (eg /tmp/opendias_test/) so that no root access is required for testing (or local deployment).
The idea is to take all location config from the GNU automake "--prefix" variable.

openDIAS version 0.8.1 released today.

The openDIAS development team are pleased to announce that openDIAS; the open source document imaging and archive system, have today released version 0.8 of the application.

OpenDIAS is a product designed to allow the home or office user to store and catalog documents, mainly (but not exclusively) from scanned physical documents. Documents are stored on a central machine and can be scanned/ imported into the archive, searched for by OCR content, date, title or assigned tag from any machine on the network.

Opendias 0.8.1 Document Detail

New features in the upcoming 0.8 release include:

• Document linkage to other docs,
• Scanning device locking,
• Document list has been migrated to an auto loaded list,
• Imported PDF, ODF and Image objects now generate a thumbnail and have OCR performed,
• Introduction of a localisation framework (initially with English and German lang packs),
• Various other frontend and backend improvements, tweaks and optimisations,
• Better testing of various hardware and OS configs (including 64-bit, Ubuntu, Redhat, Debian).

OpenDIAS is available in deb and rpm packages, from a source tarball or GitHub, with a comprehensive testing suite.

Opendias 0.8.1 Document List Page

We encourage all users to upgrade their production systems to this newest version.

For further information or help, visit the project homepage at http://opendias.essentialcollections.co.uk/

Opendias 0.8.1 Document Scanning

The next cycle begins

Work has started on the new development branch (the 0.9 cycle).

Already started is the 'configurable install location' and 'access controls by username/password'.

No rest...

Version 0.8.1 Released for public beta

The openDIAS development team are pleased to announce that openDIAS; the open source document imaging and archive system, have today released version 0.8 of the application as a public beta.

OpenDIAS is available in deb and rpm packages, from a source tarball or GitHub, with a comprehensive testing suite.

Opendias 0.8.1 Document List Page

We invite all users, either new or familiar with openDIAS to install (onto non production environments) and give feedback before the scheduled release date of Monday the 19th November 2012.

For further information or help, visit the project homepage at http://opendias.essentialcollections.co.uk/

0.8 alpha release has been frozen

The 0.8 alpha release has been frozen.


We expect the release to be final before the end of the year.

Take time to re-focus efforts

I've recently pulled back from the project to get a little perspective. I think for the past few months, things were getting carried away with updates and changes just for sake of them rather than to fulfill the big picture. I'm highly aware that people are waiting for the latest version, and the longer things just trundle on, the longer work that has already been done get waisted.

To that end, I've taken the opertunity to re-focus on generating the next release. I'll be posting shortly with the plan to get to just that point.

Done client side testing (plus test)

OK, so we have a fair smattering of client side testing now. Does what is says on the tin. No great shakes, but extra coverage is always good coverage. Unfortunalty were not going to ensure that the layout is not busted on various browsers, but still.

Client side testing

I've had opertunity recently, to experiment with various means of website client-side functionality testing. The one that's far out in front, at least for testing openDIAS is concerned, is QUnit. Our app uses a lot of JQuery for frontend functionality, so using their own testing framework to help test openDIAS looks to fit really well.

Best of all, this can be intergrated into TestSwarm (and further Jenkins), to allow testing over multiple clients on multiple devices: Linux, Mac, Pads, Phones and even (if I dust off that VM) windows.

I'm going to generate a test suit that will intergrate into the current regression tests, and will fill the 'client functionality testing gap'.

English language pack created

The moving of English text out of the application was more involved that I'd though. Not that it was difficult. It was just there was more text than I'd thought.

Anyway, we now have an EN language pack.

Efforts have started to translate this pack into German, but since I don't know German and am mainly relying on 'Google translate', this is slow going. Thankfully this is a task that can run in the background.

Finished the localisation framework

Well, that was much easier than I'd imagine it would be. The end product is almost exactly how it was describe in my earlier post. We just need to move all the current 'hard coded' English test into a language pack and replace with the framework variables.

Then it's time to localise the EN language pack in to some suitable languages - Gulp!

Experiment to create a localisation framework

As an experiment, I'm looking into how much work is required to add a localisation framework. The idea being that the app web frontend will allow the user to specify a language (from allowed 'installed languages', or leave it to the browser client to negotiate the best language depending on users machine settings. The available languages will be negotiated, depending on the backend installed 'language packs', default to English if all options fail.

Library Updates

The new version of Ubuntu does not support tesseract v2.The OCR libraries have therefore been updated to the latests version. This, and the enevitable following refactor updated several other key elements of openDIAS.

• Leptonica has replaced libtiff and freeimage, just because that's the one that's intergrated into tesseract, and it can be used for all the other image processing we are doing.

• Poppler has been introduced to parse PDF files. We can now get thumbnail images for PDFs as well as imported images and scanned docs. A temporary hook has been put in place that will allow PDFs that have already been imported to be re-parsed to get the accurate OCR test and thumbnail image.

Now 64-bit capable

Steve Meifert's changes have fixed various issues seen on 64 bit systems.

Big Thanks to Steve for his input.

UTF8 FTW

Tests have been added to ensure that UTF8 data, posted to the app, or generated by OCR is stored and rendered correctly by the API and application frontend.

I thought, I may have some trouble with this, but it looks like everything is in order and no (only very minimal) changes are required. This opens the door for full multi-language support, that is planned for the future.

Frontend Timeout

What has long been a source of problem is the way that the frontend handles error. If the API call to the backend timeouts, or worse, the backend has totally died, the frontend would just sit there.

Updates have now been added that will catch AJAX/API timeouts/errors and let the user know in as an informative way as possible.

This had a fair impact on testing, to handle slow hosts, but I think things are covered now. I'm sure things will highlight themselves in the coming months if any problems remain.

Sane has errors

So, I've started on the move about of SANE calls as described in an earlier post. However, I'm still getting problems in some simple scenarios.  I've isolated these out into their own very simple program so they can be demonstrated.

• https://github.com/clearscene/opendias/blob/d5245ca30ee4cbd7bc78bf8fa287debfe16944b3/contrib/getScannersTest.c
• https://github.com/clearscene/opendias/blob/d5245ca30ee4cbd7bc78bf8fa287debfe16944b3/contrib/makeGetScannersText

And as a result, I've raised a new bug against the sane project.

Sane Threads Fail

This weekend I decided to run the new test suite, across a 'range of machines' and all my 'testing OS's virtual machines'. These test runs showed a number of failures - segfaults on exit. Investigations showed that this is related to the calling of sane functions while within an httpd thread.

I knew that sane is not thread safe, but I thought I'd be OK if I did a 'sane_init ... sane work ... sane_exit', all within a single thread. But alas not.

So, I'm going to have to do a bit of refactoring to solve this problem. The current plan is to:

• Move the sane_init and sane_exit in the main startup/shutdown methods.
• Create a 'command socket' within the main block of the program.
• Isolate sane activity from the current methods that do sane work and replace them with calls into the command socket.
• Use a listening loop on the command socket to dispatch sane work - that will be done within the main part of the program (and not a thread).

Hopefully this will solve the problems, and has the added advantage of making 'sane and/or device' locking easier to implement. Therefore I'll be doing both of these at the same time.

Testing is back up to scratch

The marathon that is 'better.testing' is over!

The old testing was based mainly around using the frontend to exercise the backend. This was a logical approach, but was flawed in a few ways.

• First, it left little room for edge case testing - 'happy path' testing does not find issues.
• Second, using the frontend and a valgrind run backend was painfully slow.

So, I've updating things a bit. First the current tests have been split into 'service starting/stopping scenarios', and the 'current frontend tests' (with valgrind checking removed).

A new set of tests have been created, that fire HTTP request at the services just as an AJAX request would do. The responses are checked for what is expected and the database compared to ensure the action have been performed correctly. These tests are also checked under valgrind, to give confidence with regard to memory violations. 

This changes not only cover a greater range, but also allows greater control for the tests we have and flexibility for new testing in the future. 

Will be available in the 0.8 releases.

Swiftly moving on.

The document linking work sent really well, and I've got something pushed (to dev) that I'm fairly happy with. (document linking will be available in the 0.8 releases.)
While that settles in my mind, I've started on the 'better testing' branch - Loooong overdue.

Started work on 'document linking'

Created development branches and started the initial work for the document linking solution. Things are going well so far.

Wow - It's finally here - 0.7.3.

Wow !

It's finally here - 0.7.3.
Go get it people, enjoy:

• Issue #10 - Scan images display is not permission checked
• Remove the de-skew functionality
• Improve OCR operations accuracy
• Allow documents to be opened full screen
• Revamp doc list and filter functionality
• Added and 'action required' flag to documents
• Added an "ideas" link and fixed typos/spelling error, on the application homepage.
• Removed the dependency on glib
• Added new admin app to easily set config options
• Improved the selection and assignment of tags
• Fix an issue with bad setting of SANE options. (some user could not set the resolution)
• Added language selection for OCR operations
• Fix install problem if installation is already found
• Improved build instructions
• Fix build problem on Fedora
• Layed the groundwork for device locking
• Various tidy-ups
• Unified the backed response structure and defined an API

New Project Website

The new openDias project portal website has been uploaded.
Check it out : http://opendias.essentialcollections.co.uk/

Release 0.7 out for testing

All open issues for the 0.7 branch have now been closed and I've tagged the codebase (0.7.2).
Testers are giving the new release a thrashing, so the full public release will be cut shortly.