Index - Software Engineering

• Example of sensitivity analysis (July 09, 2011)
• Design Patterns for Prolog (June 25, 2010)
• Creating, Sharing, Discarding (March 14, 2010)
• Servlets and Jersey (March 01, 2010)
• Java Persistence API (JPA) (February 26, 2010)
• JSON (February 26, 2010)
• I like to model things, but... is modelling a thing of the past? (March 24, 2007)
• Implications of Web 2.0 on software architectures (March 18, 2007)
• Software deployment : Environment variables & Jar locations (March 18, 2007)
• Database connection pooling and clustered Apache Tomcats (January 27, 2007)
• Displaying feedback from the server on JSP pages (December 18, 2006)
• Levels of authorization in web applications (December 02, 2006)
• Inter-Servlet Object communication (November 25, 2006)

July 09, 2011

Example of sensitivity analysis

'Going through some old notes. Found a good tidbit 'I want to remember. 'Putting it here.

In the context of process modelling, you do a sensitivity analysis to get a sense of how your model reacts to itself in various combinations of inputs. You do a calibration to make sure your model will predict according to what your actual data says about the way things should happen.

Here's what I wrote down: A possible sensitivity analysis - Take an agent population and compare the percentage time spent in a certain state when the simulation has 1 agent vs 2 agents vs multiple agents, etc.

Another possible sensitivity analysis: Compare a lockstep simulated curriculum vs a self-directed curriculum.

Posted by Frozone Permalink on July 09, 2011 11:59 AM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

June 25, 2010

Design Patterns for Prolog

Over lunch, I was talking with some friends about design patterns and how some of them relate to specific programming language paradigms, such as object oriented or functional. (Hello to my friends, Carrie, Nikolai, Mayya & Collene!)

Before resuming my work after lunch I quickly Googled and found a paper, 'thought this was awesome:

(PDF) Patterns for Prolog Programming by Leon Sterling
http://www.springerlink.com/content/qq939j1y1c57hea2/fulltext.pdf

I believe this must be the same author as the Sterling & Shapiro's The Art of Prolog.

Posted by Frozone Permalink on June 25, 2010 01:09 PM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

March 14, 2010

Creating, Sharing, Discarding

This post is about finding the balance between "drawing inward to create" versus "putting the idea outward for changing by others".

Today, I had only 1 meeting. So, I spent nearly all day at my desk writing code. At the end of the day I was surprised in two ways.
1) joyfully, that I made good progress in building something that could help many people one day
2) I felt guilt from a day lost of writing reports, fostering professional relationships and making necessary connections. Possibilities unexplored.

As I grow into a strong leader with technical aptitude, given a project I must decide *how* to guide its growth: technically or broad-directionally. If you spend time connecting with others, the project will refine its overall direction, getting more specific in its purpose. However, too much talking and zero code-writing results in zero progress. If you spend time in system design, you uncover important questions about how to build the thing such that it is flexible, scalable. Also, when writing code, you commit pretty heavily in a particular direction (given the value of the time that was spent writing that code).

Both are good. I am exploring the balance right now, choosing when to commit direction while remaining nimble to constructive criticism from others.

Also, reflecting on the sheer experience of getting lost in design again: In many ways I had forgotten how good I was, and how quickly I could produce beautiful, powerful design. And in the same day, I frightened myself because I saw my uncertainties and I made design decisions without theoretical backing (i.e. without reference to a design pattern). I remembered how many years it has taken me to get where I am today. And where I saw room for improvement in my work, I felt fear because I understand about how much more work it would take for me to evolve further in a given direction. And I have a rough idea of a human lifespan, and would dare to guess at the limits of my own lifespan. And I felt finite because I see that the oceans of possibility for my growth, while in my hands I have only a tasse of water that is my life, and I must choose harshly, for the ocean is bigger than the tasse.

heh. I am so introspective it is almost cheesy.

But reflection is key to evolution.

Posted by Frozone Permalink on March 14, 2010 10:29 AM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

March 01, 2010

Servlets and Jersey

What's the difference between a Jersey service and implementing my own Servlet?

(I will post the answer here when I learn it.)

Posted by Frozone Permalink on March 01, 2010 10:58 AM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

February 26, 2010

Java Persistence API (JPA)

The Java Persistence API, (JPA) seems to have formalized the custom domain modeling and querying I used to do.

I am starting to see how all of this hot new stuff is supporting and formalizing in a scalable way what used to be a custom job.

This is great, but the only way for me to overcome the scariness of these new tools and leverage them so that they are actually making me *more* productive is to roll up my sleeves and write a core, trial system. There is no way I'm going to have desk time for that on Monday, Tuesday or Wednesday of next week. So I am looking at Thursday - pretty much a week from now - before I can start. I had better be pretty certain that this is the way I want to go. I think I am. It's familiar territory, for the most part.

I find that the more I age, the more I rely on my past experience.

Posted by Frozone Permalink on February 26, 2010 03:42 PM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

JSON

Basically, JSON is an alternative to XML.

Posted by Frozone Permalink on February 26, 2010 02:41 PM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

March 24, 2007

I like to model things, but... is modelling a thing of the past?

Continuing my thoughts from my previous entry,Implications of Web 2.0 on software architectures,: I like to model things, but is domain-modelling a practice that will become obsolete? I'm worried about how manual my software is -- I mean, it just takes so much love and care and attention to build a decent piece of software -- and I'm wondering which aspects of the software development cycle will be subsumed by a combination of automated ontology-discovery, ontology-generation and social networking.

I mean, the OO-programmers of the world have already modelled and re-modelled the same things over and over and I think that's probably one reason why web-ontologies were born: to try and co-operate and be able re-use the same models from application to application. (Right? Oh, gosh, I feel so lost. And I worry too much. lol )

By "modelling", I'm talking about what you learn in first-year Computer Science. For example, to model a cat, you would define a class "Cat" and list several instance variables of different data types with getters/setters. Your Cat may have a private String name; a private License lic (where License is another object that has also been modeleld), a Breed b, etc. etc.

You would also write functions into the Cat that correspond to the real-world behaviour that you'd like to have happen in your application. For example, Cats may take in a Tree object and perform a hunt() operation on on that Tree. The hunt() method could then return another object of type DeadBird in the process (if your Cat is anything like ma jolie chatton Minou!).

Or, maybe instead of placing the method hunt() inside of the Cat class, you would have a Controller class listen for the state-change that would allow the user to request a hunt() by their Cat and then offload the Cat object and the Tree object into some HuntEnvironment simulation and then wait for the simulation to return both your Cat and your Tree back to you after they have been modified by the happenings in the Environment. Decisions on where to put these methods is the art of software engineering, and is what I spend a good portion of my everyday job doing: Organizing, assigning responsibilities, modelling and making these decisions. It's very abstract work and is hard to explain. A lot of people will tell you that it isn't worthwhile, but I try to keep my head up. I believe in AI, in engineering, in learning, in evolution and in people. It's just hard to keep one's purpose and one's values in good order amongst all the craziness!

This is one reason why I'm sitting here and scratching my head, wondering about the future. When will human-made models be relpaced by computer-made models? And how do I work with that in my software architectures?

I thought: Well, maybe the impact of computers-doing-the-modelling-work (ontology-generation and ontology-discovery) will impact me as a human software developer in phases. The first obvious phase is that the task domain objects will be taken over. Like, "the things a person can learn about": concepts, understanding and just the hard representation of knowledge -- curriculum-type things.

What about business-model things? Will human programmers forever be modelling things like "Person" or "Student" or "Animal" or "Product" or "Employee" or "Department" or "Course"? I'd imagine that after a while some of these things would just be so repetative that you just have to have a standardized model somewhere, that could be adapted and would maybe even evolve, depending on how it needs to be used in a specific application. Maybe there will be a bazillon ways to model the object Employee, but after a certain period of time we'll find that many attributes will last (like FirstName, Nickname) while other attributes may get killed off in an evolutionary manner.

What about the objects running control-procedures, specific database operations and user-interfacting? These are so icky-picky, especially with technology in 2007 we are probably a long ways away from saving human programmers that kind of work. After the automation of domain objects, maybe user-models are next.

In the short-term, I would like to learn more about referencing ontologies in my applications, something I haven't worked on very much since last July.

It is a strange, strange world -- and it is completely different to each person who looks at it. What do other people worry about?

Posted by Frozone Permalink on March 24, 2007 01:46 PM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

March 18, 2007

Implications of Web 2.0 on software architectures

Web 2.0 - this web-based software that "gets better as more people use it" has got me thinking about some of the systems I've designed in the past and really wondering if being a web developer in the Web 2.0 world means that I have to face a fundamental change in my software architectures. Or, is Web 2.0 is just a fad that grows because popular understaning of web-based technologies has finally caught up with reality*, and is just a change of human-perception of technology? (*Reality, as I somewhat snobbishly think of it, being "what has been going on in the academic world for years and years". And I'm not even a grad student yet! lol )

In other words, part of me is thinking: "Technology has been evolving in this direction for a long time, what's the fuss about? Did something magical happen that I did not know about, and should I be going back to school or something in order to catch up?" Or, even deeper: "Are things changing faster than I'm really aware of, and is my skill-set adapatable enough that I will be able to keep up?"

My brain is wired for Model-View-Controller -- most of the stuff I build comes out in this shape. Can I keep working in this comfortable format, just thinking of the "Model" as a shared thing that lots of people will be updating, rather than just a select few? Architecturally-speaking, my Models are usually sets of modelled Domain Objects that can be juggled around in tasks - mostly to give feedback to the user as they navigate around and figure out what they heck they want to do. Occasionally they do wish to make a change, so the domain objects & appropriate back-end database are updated in a nice MVC manner. Will Web 2.0 still have these rather large and complex server-side Models of the task domain? Or, will Web 2.0 mean many, smaller & distributed models? (ex. Ontologies in OWL)

Hmmm, I don't know! Or maybe I will read my own blog entry in a month or two and will wonder why I was so behind-the-times in my worryings. :-P

Posted by Frozone Permalink on March 18, 2007 06:01 PM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

Software deployment : Environment variables & Jar locations

This will probably seem obvious, but it's a lesson I learned from experience: When you deploy your software, be sure to bundle your own .jars and your dependencies such that they are NOT inside your JAVA_HOME nor are they inside your TOMCAT_HOME folders. Why? Because Javas and (less frequently) Tomcats may be automatically updated without your knowledge, and consequently the hard paths that were pointed to by environment variables JAVA_HOME and TOMCAT_HOME will get changed after the version upgrades, leaving your poor Jars left behind in forgotten & out-of-date file folders. This can cause your app to suddenly and unexpectedly throw java.lang.ClassNotFoundException s.

I openly admit that I don't know the best tricks of the trade to deal with this situation. Maybe create a new environment variable called MY_APPLICATION_HOME and then just be sure that your CLASSPATH includes it?

Or maybe I'm so behind-the-times that there are tools out there nowdays to deal with these things and I'm worried about something that should actually be handled by robots.

Semi-related article on ONJava.com: Managing Your Dependencies with JDepend

Posted by Frozone Permalink on March 18, 2007 04:58 PM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

January 27, 2007

Database connection pooling and clustered Apache Tomcats

As much as I enjoy/prefer theoretcial work, sometimes reality sets in and you've just got to roll up your sleeves and clean it up!

Here's the situation:

I've got 3 Apache Tomcats running on port 8080 (behind our organization's firewall) on separate machines. They are load-balanced using mod_jk/1.2.14 behind an Apache HTTP server. Thus, the only way to interact with the Tomcats (from outside the firewall, anyway) is via HTTP traffic on port 80 on the web server.

The application requires access to 5 databases. Each is hosted by a different machine. (Some of these machines are the the same as the Tomcats.)

Because I don't know any better, I created the 5 ODBC connections under "Control Panel/AdministrativeTools/ODBCAdministrator" on *each* of the Tomcat machines. How else can an application running in a Tomcat Servlet access a remote database? I don't know, and maybe there is a better option out there for me; perhaps in the form of a different protocol. But in the meantime, this is working, at least.

My problem: Lingering database locks. If TomcatA is using a database, and if the application doesn't give up the lock, and if a user whose session is being served by TomcatB tries to read information from the same database, then I get an error (was it java.sql.SQLException or java.lang.NullPointerException? I can't remember; 'should have written it down.)

Obviously, I have to clean up the lock on TomcatA (Thinking, "I explicitly told it to use a non-exclusive lock, why is it doing this?!"). Secondly, I got thinking about the whens and wheres of my database reads: When do I have to open the connections, and when do I close them? Sometimes, if you have to do multiple queries in sequence, it's better to re-use the same connection for those 3 queries. Other times, it's best to swoop in, grab your data, then close it ASAP so that the many users on one of the other Tomcats get a turn.

In a multi-user system, I thought I could be all smart and use static Connection objects, so that each Tomcat would have only 1 connection shared by all its users, thus eliminating the possiblity of 2 connections clashing with each other (at least on the same Tomacat). The Connection would be closed if nobody was using it, and opened so that a handful of users could use the same Connection to execute their Satements. This approach didn't work, because if "you", as the application, are encapsulated within a user's Session, you have absolutely no way of knowing if anybody else is using the Connection, or will ever need that Connection, so who are you to decide whether it should be opened or closed? This doesn't help the lingering-lock problem, either. Further, there were no brains for sharing the database connections between multiple Tomcats. So, I scrapped that approach.

(In my own defense, I came up with the aforementioned approach in 2004, so I'm actually much smarter now. Really! heh heh)

Later, I discovered database connection pooling. This approach takes the decision of when to open/close the "master" connection away from me as a programmer and lets the Tomcat handle it. Yay!

Now that I've been working in a connection-pooled environment for a while (though I'm still suspicious about whether or not I've got it all hooked up properly) I'm starting to develop some questions:

Are there any brains for co-ordinating multiple-Tomcat access to a database?

Does "having an open Connection" always nececessarily mean that you have it locked?

All 5 databases in this system are in Microsoft Access format. (Not my choice.) Are some database-locking functions (ex. per-record vs. per-database, exclusive vs. non-exclusive) dependent on the database implmenetation? Is that why I'm having so many problems trying to obtain multiple reads from multiple servers at the same time?

Is a Connection created on a per-user basis? or a per-Servlet basis? or a per-Tomcat basis? i.e. If I have 2 users requiring access to the same database, are there 2 connections for the same database inserted into the pool? Or, is perhaps the same connection used, and it's just that the 2 users are able to get the same connection by way of that pool? I don't see the big picture yet.

How about per-record locking, as opposed to per-database or per-table locking? Can 2 users have the same database open at the same time if they are operating on different records? (Obviously this is true -- I know this from experience from a different project that used a MySQL database. That system is MUCH more stable.)

When I explicitly do a "Connection.close()", I get connection pool-related errors (shoot, 'don't have the wording with me). Does this mean that I'm doing my "close()" from the wrong place, and is that why I have this lingering-lock problem?

Do I still have to worry about the problem of handling the use of the same Connection over multiple Statements? Or should I always just do the Connection.close() at the end of every single Statement even if I know I'm going to need the same connection again right away, and let the pooling function handle the management?

Why am I doing this?

lol. I have about 4500 users -- maybe 1 Tomcat is enough...? With fewer contestants vying for these valuable Connections/Locks, then maybe if I have only 1 Tomcat then my lingering-lock problem can be swept under the rug because I only have 1 Tomcat that needs the lock anyway.

When I -am- faced with an out-of-control database lock, why doesn't re-starting my Tomcat service release that lock?

And why is the entire database *locked* when I explicitly programmed in a non-exclusive connection?

In Microsoft Access, is there even a difference between "having an Open Connection" and "having the Database Locked" ? According to my expericence, the answer is "No" but I'm still trying to figure all of this out and I'm probably wrong.

I think I am starting to repeat some questions here, so I'll quit. :-)

Well................. time to go do the laundry and make some banana bread. Bye!

Posted by Frozone Permalink on January 27, 2007 10:19 AM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

December 18, 2006

Displaying feedback from the server on JSP pages

I had to dig out my second-year textbook on digital logic to solve this one.

Going back to the problem I had been considering about the different "states" of an application and how this transposed onto a Servlet / jsp-pages implementation, I was thinking about how each individual jsp page is sort of like a "state" and that the Servlet is the master controller to take some input and then determine the next stat that the user gets sent to.

I couldn't remember how to approach the problem, but my trusty textbook - in Chapter 8, Section 1: 8.1: Basic Design Steps - the authors guided me through the process of listing the set of possible states, and then figuring out which conditions allow the user to flow around through the different states.

In the end, I'm satisfied with my design that each jsp page is a different "state" or "mode" or "step" or whatever in the big picture of the application, and that each jsp page is its own automaton with its own individual set of inputs and outputs (i.e. HTTP GET variables.). The server can send error codes back to the jsp page to tell it which sort of message it should display to the user, and, the user can send inputs within the jsp page to tell the server what action (s)he wants to make within the application.

Posted by Frozone Permalink on December 18, 2006 11:38 AM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

December 02, 2006

Levels of authorization in web applications

Most of the applications I've built in the past have only 2 levels of authorization:

(1) You're logged in and have access to various functions according to your "group" or "access" memberships
(2) You're not logged in, and therefore have access to nothing.

My latest project would benefit from having some functions available to everyone - even those unauthenticated. It is basically an item-selection service; certain items are offered by certain bodies, and these items are not secret and in fact would be great if they were publicly-available. The only part that needs to be protected is for a given user to be able to modify their own item-selections (and also protect what those item-selections *are*). The items are sufficiently complex that navigation through them could be made as a stand-alone application, even without any functionality for "picking" them.

(Note: In the description above, I had previously worded "item" as "service" but then made the change because I didn't want to cause confusion with my earlier notes about WSDL.)

I was thinking about a Servlet's objects -- maybe some are static to the Servlet class, and others vary per HTTP request -- and also about how these relate to a user's session variables and also the values that can be attached to any HTTP request. Some of the business logic -- the public stuff -- is available as a static attribute of my Servlet, where all users can refer to the same "copy" of the item being offered, thereby allowing the system to be aware of how many people want a particular object via memory references. No database query needed; the system just "knows".

I guess pretty much all domain objects live at the servlet level except for a user's individual navigational and usage objects; even these may ultimately refer to their platonic ideal "singular" within the Servlet.

What -are- session variables supposed to be used for, anyway? Navigational cues? Application-data that is reflected in the back-end database? I've always assumed that the session variables should be "the stuff that the user is working on and manipulating right now" but I am such a newbie I could be wrong and not even know it!

And, on top of all of this, I have people telling me that I am wasting my time thinking about object-modelling and that I should only care if the program works or not. The attitude is like, "the more stuff you build, the more familliar you get with the tools and the faster you can build more stuff." Then, I was mildly insulted when my co-worker insinuated that because academics aren't charged with "building stuff" that they can't possibly learn the best way to do so, and therefore no worthwhile learning can occur in a non-industrial setting. Uragh!

Personally, I have found that having domain knowledge as actualized objects-in-memory is necessary for intelligent rationalization (for example: arranging the objects in a graph and then performing a shortest-paths traversal on it), as opposed to a no-modelling approach of merely brute-force querying a database each time object-data is needed.

'Apologies for the rant. I considered deleting it, but, meh. I am a human and am entitled to my bouts of irrational emotional turmoil.

UPDATE: I found a related design pattern - Core J2EE Patterns - Intercepting Filter. I like this pattern because some of my clients are asking for the system to authenticate against an LDAP directory, while other clients want the system to authenticate against an internal database where the password is stored as a field in one of the tables. This design pattern will allow me to write the components for both, then I can just plug/unplug each filter according to what the client wants. I wonder if I can eventually apply different combinations of these filters to different parts of the system. Then, with no fileter, users can browse around the "free" stuff that does not require authentication. As they go deeper into the system, extra authentication filters can be added. I guess. Maybe. Meh.

Posted by Frozone Permalink on December 02, 2006 09:15 AM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this:

November 25, 2006

Inter-Servlet Object communication

I've got two servlets. One is an expert in one domain -- take as an example, bicycles -- and the other servlet is an expert in another domain -- say, livestock. You know: cows, sheep, pigs. (Oh dear, I can see already that this is going to turn into a dumb example. Oh well.)

Anyway, suppose that BicycleServlet wishes to survey the relative speed of one of its model bicycles with other moving objects in the world so it can advertise its speed relative to those other things. 'Maybe for targeting specific markets. Whatever. Suppose, then, that it decides it would like to see whether or not the token bicycle is faster than a rampaging herd of sheep.

(Ugh, did I just contrive that example?)

So, BicycleServlet must contact LivestockServlet to grab some specialized information about sheep and their behaviour under certain conditions to see how fast they would go; then it can complete its computational comparisons with its own bicycle.

Maybe BicycleServlet has its own rough idea of what a sheep is:

So, maybe it sends its sketchy sheep object to LivestockServlet saying "give me more information on one of these".

LivestockServlet, then, has a much better idea of a sheep, full of detail:

So, it packages up the data and fires it back to BicycleServlet, who happily continues with its computation.

I attempted a similar servlet-to-servlet object swap (in a different domain, obviously) and it's working quite well. However, I'm unsatisifed with my design because it severely limits the flexibility of inter-servlet communication. For instance:

BicycleServlet
doGet() - empty (at least, nothing relevant to this example)
doPost() - (1) send sketchySheep; (4) receive realSheep from LivestockServlet

LivestockServlet
doGet() - empty
doPost() - (2)receive sketchySheep; (3) retrieve realSheep; (4)send realSheep;

I'm thinking to myself, "Surely, in the future, LivestockServlet will have better things to do than to exist solely for the purposes of serving sheep to the BicycleServlet."

Without question, we need a protocol.

A little voice in the back of my head shouted, "WSDL!" but I admit that I don't really understand the Web Services Description Language, what it is, how it applies, and how it can solve my problem.

Maybe I'll go work through those tutorials to give me an idea -- maybe there is a better way of requesting information from a server than sending off a sketchy sheep.

Posted by Frozone Permalink on November 25, 2006 09:33 AM | Comments (0)
categorized under Software Engineering

	Tweet
This entry was written under the moon phase that looks like this: