User Experience, Design Choices, and Learning Analytics: A lesson learned with ProSolo

Design choices made in different phases of the development of a socio-technical system may have implications on user experience that are not easily anticipated. Our intentions have always been to provide the users of ProSolo with the best possible experience. However, one such design decision turned out to require further consideration after a DALMOOC participant reported an issue with the automatic logout from ProSolo. Rightfully, the user wanted better experience as being logged out automatically can be annoying. This was caused by our decision to set the time of automatic logout after 10 minutes of inactivity. The reasons for us making this decision were multi-fold, and here we touch on the three major ones – security of user data, use of servers and associated environmental impact, and accuracy of learning analytics. With this, we would like to reach out to the ProSolo users and ask for their input how to approach this issue and balance these three factors.

Being logged in to any Web application when losing either a laptop or mobile phone can pose a serious threat to the security of user data. Not to mention the potential abuse of the user identity! The user just needs that inconvenience in addition of the loss of their favorite device. Similarly, the use of a personal computer creates a habit that many don’t logout from any application. Why would we, as that’s another thing to type in? But, forgetting to logout, while working at a public computer (e.g. lab), can again create a similar data security threat. In all these situations, it is ultimately the user who personally gets affected with the decision to have a long timeout (or no timeout at) without affecting  others with that decision. And, who is the software development group to make that choice on behalf of the user, isn’t?!  Thus, from the software development, this case is easiest to deal with and simply to follow fully the user choice whatever that be. Is that ideal?

Cost and environmental implications are connected with an increased use of servers if users are not automatically logged out after some period of inactivity. If a user is not active, or they have forgotten to close a window in which the application was open, the servers will still be allocated for that user. While it might not be visible, any application such as ProSolo has a distributed server architecture. If one server gets too busy when serving several users, a new server needs to be initiated (through the process known as auto-scaling) in order to serve all the users. If users stay logged in, we need more servers and that increases the cost of hosting. For the volume of the use during DALMOOC and the generous support of UT Arlington for the hosting servers, this is not a major concern for ProSolo at this point. Even with the growth of the user base, the financial resources can be found. However, we need not to forget the potential environmental impact – that is, the use of additional servers means more energy spent. If the energy does not come from green sources (as common today, unfortunately), that’s not a desirable outcome we would like to see. For the moment, similar to the costs, that’s not the major issue for ProSolo given the amount of resources needed. It will soon become we hope, especially, if we want to promote environmentally friendly technology – which is our goal! In this case, the user experience may also have broader implications than just security and becomes a question how to balance the user experience needs and responsible energy consumption?

Learning analytics are less accurate if we don’t know when users become inactive and logged out. Together with Vitomir Kovanovic, Srecko Joksimovic and Ryan Baker, members of the ProSolo team Shane Dawson and Dragan Gasevic have recently submitted a paper to the LAK’15 conference. In the paper, they analyzed different problems of estimation of time users spend online in learning technologies like LMSs (btw – we are happy to share the paper draft, if there is interest). When we have a clear logout event logged with a learning tool, as originally used in ProSolo, the estimation of time becomes much more accurate. Otherwise, different strategies for estimation of time spent online are needed (in the mentioned paper, 15 strategies were analyzed). The choice of those strategies in the estimation of time on task can have a big impact on the results of learning analytics. The above study showed that the differences in the explained variability of regression models – gauging learning success based on time spent online – can be as much as 15%. That is, a regression model with one time-online estimation strategy explained about 45% in the variability of the learning outcome (the student grades), while another one only 30%. An even bigger problem is that we don’t know – without further observational studies – which estimation strategy is best (or optimal).

The above research led us to the decision to introduce the timeout of 10 minutes to increase the accuracy of the measurement of time spent actively using ProSolo. Informed by the existing research, we thought that this would not hurt user experience. As turned out, we were not quite correct! This time for automatic logout should have led to the two major benefits learning analytics research: i) we would know which estimation strategy was the most appropriate to use when/if we even remove mandatory timeout; and ii) any analytics performed based on the data collected during this course will be much more accurate. Although the benefits of more accurate analytics should improve user experience in the long run (why else working on learning analytics?!), reasons of this nature cannot easily be justified and should not be given a priority over the user experience.  Alternative study designs for time estimation should be used instead!

We hope that this post sheds some lights on the complexity associated with the design choices when developing a software feature that seems rather simple on the surface. As the user perspective and review is the most critical driving factor for the design and future development of ProSolo, we look forward to learning about your ideas and reflections how this and related issues should be addressed.

In the following blogs, we will discuss other lessons learned and important questions raised by DALMOOC participants and ProSolo users.

Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *