Index - Methodology

• Funnies with Timestamps (January 07, 2012)
• Building your simulation model: Identify your independent and dependent variables (November 26, 2011)
• Generative Social Science (January 24, 2011)
• Thesis topic (January 12, 2011)
• Research Methodology #2 (March 20, 2010)
• Research methods (November 01, 2009)

January 07, 2012

Funnies with Timestamps

This is my second funny experience with a simulation environment. (Here's the first one, previous entry called Simulation funnies).

I am working with the Ecological Approach, which means that each time a Learner uses a Learning Object, a "stamp" of the current Learner Model is saved on the Learning Object.

In my simulation model, I have Learners randomly visiting learning objects and leaving stamps as they go.

At one point, I have to query to do a search for a particular Learner Model Stamp. For a primary key, I had (LearnerID, LearningObjectID, timestamp). Because a student would never visit the same learning object two times in the exact same millisecond. Right? Wrong. My hotshot computer is so fast and I have a small enough number of learners and learning objects that I did actually have duplicates, so, my "primary key" was not unique after all.

That's the funny part. hahaha, get it?

Anyway, to fix it, I decided just to insert a deliberate pause after each visit to guarantee the unique timestamp. This will slow down my simulation, obviously, but this is OK because I was already getting WAY more interaction than necessary for my model.

November 26, 2011

Building your simulation model: Identify your independent and dependent variables

I just went through my simulation model and I made a long list of all my variables. Examples:

- number of learning objects (LOs)
- number of categories of LOs
- percentages of the LO population in each category
- number of learners

etc.

Then I went through and I labelled each variable as "Independent" or "Dependent". An independent variable is one where I consider it to be a "lever" that I can adjust in my simulation model. There are most likely starting values that are most realistic or most likely to happen in the real world... like a degree of "typical" or "atypical" but there is usually necessarily some variation. These are the variables where I might want to perform "parameterization" where I try to discover the best starting values for these. Other variables might need "calibration" to real usage data, where I run a simulation a zillion times and have some sort of penalty function to try and match whatever inputs to my simulation that best fit the sample data. Then I would use the result of the calibration as input to run the simulation model AGAIN, only the next time instead of running it against a real live dataset, I would set the input parameters to be whatever I want and then use the output as a way of making a very well-informed prediction.

Whew! So that's independent variables. I found that MOST of my variables are independent variables. Right now I have 12 of those.

A dependent variable is something that I am trying to measure. I have 2 of these. One of my dependent varibales is a formula. The inputs are "learning object graph & learner attributes" and "path taken by learners" and the output is the "assessment / learning outcome". If I have a whole swack of data, I can use machine learning to discover this formula. This formula is my dependent variable. My thesis statement is a claim about this variable, I think.

So.... yeah. I just wanted to note that it was really illuminating for me to say aloud that the vast majority of my variables are Independent. I have an extremely high degree of uncertainty and complexity. I need to run some thorough sensitivity analyses just to understand my baseline.

This helps me solidify the kind of research I'm doing. How does this relate to other fields? Is the characterization of my work usual or unusual?

January 24, 2011

Generative Social Science

A while ago, I questioned the hullabaloo around the computation of Nash Equilibria. (see previous blog entry). I have just found someone else who asked a similar question: Dr. Joshua Epstein, in his 2006 book, Generative Social Science. This book is one of the recommended readings for the class I am taking this term (CMPT 858: Topics in Modeling and Operations Research).

Hopefully, it will be okay for me to share an excerpt from this copyrighted book for the purpose of highlighting the question around Nash Equilibria:

...game theory and mathematical economics (the twin pillars of contemporary social science) are overwhelmingly concerned with equilibria, Nash equilibrium being the most important example. Indeed, in these quarters, "explaining an observed social pattern" is basically understood to mean "demonstrating that it is the Nash equilibrium ... of some game". However, these are mere demonstrations of existence. Per se, they do not demonstrate that the configurations of interest -the patterns allegedly explained - are attainable at all... (from the Introduction, page xii)

The author presents their generative methodology as a more appropriate and more powerful tool to tackle the problem. I am really, really looking forward to reading the rest of this book!

January 12, 2011

Thesis topic

Well, folks: it's all starting to come together into a coherent project idea. I have my thesis topic!

I am going to build an eLearning system that can automatically incorporate new materials into the course.

Imagine you're learning something new, and that you have gathered a set of web pages that help you learn about the topic. My system will explore the ability to take a new web page (and, for it to be useful, the new web page would have something to do with the topic at hand, but this is not a requirement) and, given the starting set, it will be able to incorporate any new page in a meaningful way. Like, it will know whether the new page should be presented to a learner before pageA but after pageB, or perhaps as an alternative to pageC. My system will have built-in pedagogical structures like "is-example-of" or "is-generalization-of" to support the types of relationships between pages (and they wouldn't necessarily have to be pages; they could be youTube videos or a real-world instruction like "go take a walk to the monument in the park and think about topicX and come back with your thoughts".

The core data of my system will be in the theme of the work of my advisor, Dr. Gord McCalla, as in [McCalla 2004: The Ecological Approach to the Design of E-Learning Environments: Purpose-based Capture and Use of Information About Learners] .

For methodology, it is too early for me to know if I will be doing human subject studies or just run a simulation instead. I am leaning toward the latter, but have some work to do to convince myself that a simulation can still be meaningful (rather than just amplifying my own hypotheses!)

I have a thorn in my side about the value of Instructional Design. My thesis work will be making a lot of ID "unneccessary". However, I truly believe there is something to be said for ID. So, I have to figure out what that is, and what necessary pieces need to be built in. Right now, I'm thinking that the main point of a "course" is that it's an easy way to communicate to other people the nature of a set of learning experiences. Therefore, my system will have to have some kind of "learning objectives" built in, and having this will heal the ID-thorn in my side!

I am truly enjoying old posts on my blog like this one, from about 5 years ago. (Have I really been research blogging that long? LOL!) Re-reading that, today I know that I truly, truly have a different perspective and will be able to tackle these sorts of questions with sharper teeth.

March 20, 2010

Research Methodology #2

I have mused on this topic before.

I was just reading "What the Best College Teachers Do" by Ken Burns. I admired how they constructed their research methodology.

I am still not sure where my own work fits, quantitatively- and qualitatively-speaking.

November 01, 2009

Research methods

I want to do a study about instructional planning. It's hard to know where to start because I can't pinpoint what, exactly, it is about instructional planning that I want to create, to measure, to experiment with, to optimize... Nor do I know very much about conducting proper scientific studies. Well, that's not exactly true. I know quite a lot about science from general interest reading across the disciplines (cognitive science, psychology, sociology, medicine, optimizing algorithms...). I just want to note that I've never actually conducted a study myself.

I thought that this page was useful, from Birkbeck University of London. From this, I could kind of list or sketch out different types of research methodologies. (Simulations, case studies, empirical studies, proofs...).

I'm struggling a lot because I don't have an advisor with whom I can talk about these things. Sure, I have my friends who have done grad studies before, and I can surf around and read the blog entries of other students, or read "How To Be a Successful Grad Student" type articles written by professors, but it's not the same as having a real person or persons who know your work, whose job it is to talk to you about your work, and can share their wisdom with you. I am craving mentorship.

I think I'm going in the right direction... I mean, I'm working on my application for grad school by dusting off the ol' resume, planning to approach some professors for reference letters.... It just seems so FAR AWAY right now to me. I want to do research, but I feel like I have to jump through a million hoops before I can actually get down to business. The barriers seem very high to me. Perhaps I'm just feeling sorry for myself! Part of the problem is that I have very little (okay, zero) computer time at home, and I just have to get better at using my lunch hours and coffee breaks at work to get this stuff done.

Anyway, getting back on topic: Research methods. In the area of planning, your planning algorithm is successful if your agent reaches its goal while performing the fewest number of steps. This measure (the number of steps taken by the agent) isn't the right measure for my area. For instance, if an instructional planner is guiding a student through a guided exploration of some course material, throwing in some games or exercises here and there, it doesn't matter so much how many "steps" it takes for the student to "understand". I've talked about this and have put my thumb on this before: Utility, in my problem, is meaningfulness of the user's experience. (See related entries: My friend, the utility function, and This dichotomy)

So, is this an empirical study? Or am I doing qualitative research? Probably some of both.

I guess I really do need to define my problem better before I can design a study around it!