The themes discovered from the literature were often clear and easy to establish, and many texts reiterated similar topics and concepts. Whilst some distinct ‘ambivalence’ existed in perspective or interpretation by some of the researchers in the texts, this was in itself a point of interest for this research, as shed light on theme context and placement into the PBH (Problems and Benefits Hierarchy).
Establishing themes to build a picture of thinking and practice around uptake of technology in learning and teaching has proved a fairly functional and informative way to approach creating a basis by which a variety of additional data sources can be placed via a series of further structured combinations for comparison analysis. This then allows an attempt to draw some tentative conclusions around problems and benefits of the increase in technology uptake for learning and teaching.
Using the literature review to establish the key themes by which all other categories and topics were then adapted and aligned gave a clear system by which to accommodate all data, avoiding too much duplication or overlap, though some is always present, particularly in some themes. This is in the nature of the territory, and is therefore acknowledged as a challenge to interpretation and analysis in this pilot study.
Themes were therefore established, and a frequency table was drawn up by allocating a simple code to each text, noting mentions of any theme accordingly. This established the frequency of each theme occurrence across all literature in the review. An occurrence was regarded as some aspect of the text or paper being aligned or directly involved with the theme or themes.
In total, sixteen themes were established:
- INSTITUTION SUPPORT
- SOCIETAL CHANGES
- ICT SUPPORT (Provision)
- ELEARNING SUPPORT (Provision)
- SKILLS AND TRAINING (All aspects, Students & Staff)
- COST (All aspects)
- DEPENDENCIES (Infrastructure & Systems)
- DIGITAL DIVIDE (Inc Device Divides)
- PEDAGOGY / LEARNING DESIGN
- STUDENT CENTRED LEARNING
- LEARNING QUALITY
- CONVENIENCE / WORK LIFE BALANCE
- SHARED RESOURCES (All aspects)
- ASSUMPTIONS (idealism)
- > Click this link for Theme table with literature code allocation and frequency totals
- > Click this link for Theme Frequency totals piechart
Of the sixteen themes most often occurring in the literature, the top six most frequently appearing were Institutional Support, Societal Changes, Pedagogy/Learning Design, Student Centred Learning, Learning Quality and Convenience/WorkLife Balance.
|Theme in Literature||Number of occurrences||Percentage of occurrences|
|INSTITUTIONAL SUPPORT||9 occurrences||8%|
|SOCIETAL CHANGES||12 occurrences||10.6%|
|PEDAGOGY / LEARNING DESIGN||13 occurrences||11.5%|
|STUDENT CENTRED LEARNING||12 occurrences||10.6%|
|LEARNING QUALITY||13 occurrences||11.5%|
|CONVENIENCE / WORK LIFE BALANCE||9 occurrences||8%|
Table 1 showing frequency of top six literature themes
Themes were divided into problems, benefits, according to information from the literature review (predominantly being seen as either causing problems or benefits). Most themes were either one or the other, but a minority of themes were both problems and benefits. The placing of the theme as either a problem or a benefit was significant as would be further interpreted by data derived from the stakeholder research groups.
|Theme in Literature||Problem or Benefit (literature setting)|
|PEDAGOGY / LEARNING DESIGN||Benefits|
|STUDENT CENTRED LEARNING||Problems and Benefits|
|LEARNING QUALITY||Problems and Benefits|
|CONVENIENCE / WORK LIFE BALANCE||Benefits|
Table 2 showing problem or benefit setting of literature theme
These themes then indicated that a further level of analysis was required, that of context. By establishing not only the type and frequency but also the context of a theme’s occurrence in the literature, one might be able to establish its place in the Problems and Benefits Hierarchy more accurately. The fact that the PBH required some level of interpretation as to ‘real’, ‘imagined’, ‘intermittent’, ‘persistent’ and ‘legacy’ led to a further set of contextual categories being developed in order to give informative setting to the PBH placing. These categories sought to shed further light on how a theme occurrence had appeared in terms of what had led to it’s appearance. The contextual categories fell into four broad areas, with categories in each area. Aligning them with the PBH factors created a system of measurement of context. Categories could therefore estimate levels of ‘realness’ or ‘persistence’, for example, by looking at theme context in this way.
An interpretivist approach was used to create the context categories, as once data is seen – in this case the content of the research papers – it becomes clearer how estimations of context of themes might be measured. It appeared from the literature that using the category terms listed here would most accurately capture the type of context being dealt with. Whilst not a robust set of terms, it may be that this is in itself a finding of the research, in that an explicit set of reliable terms can be applied for the interpretation of context of research data such as being analysed here.
These context categories were then assigned to theme occurrences for each source (though not individual ‘macro’ occurrences), using a simple scale of terms, in order to indicate their consequent PBH factor. See below for terms used and consequent PBH factor, and for table containing all occurrences with their contextual category allocations.
|Contextual Scope||Types of context in scope (examples)||Description||PBH Factor|
|CONJECTURE, SPECULATION, ASSUMPTION||PA – Personal Assumption||statement from a personal or individual standpoint with no evidence or expertise present||[PBH:Imagined]|
|DC – Data Conjecture||statement being attributed to data which is not adequately evidenced|
|HS – Hearsay||statement attributed to ‘everyone’ or similar, which is only assumption|
|CONTEXT ASSOCIATION, STRENGTH AND FREQUENCY
||IAL – Contextual interpretation/association low strength & frequency||occurrence measurement of theme in research – low||[PBH:Intermittent]|
|IAH – Contextual interpretation/association high strength & frequency||occurrence measurement of theme in research – high||[PBH:Persistent]|
|EVIDENCE AND EXPERTISE||EK – Expert Knowledge||statement made from deduction or logic using expert knowledge, but not directly connected to research evidence||[PBH:Real]|
|RE – Research Evidence||statement made as a result of research evidence [PBH:Real]|
|SYSTEMS, CHANGE, THE PAST||P&C – Refers to the past or need for change||statement about the past, or requirement for change in systems or central strategies, policies or provision||[PBH:Legacy]|
Table 3 showing key to terms for literature theme contextual categories and PBH allocation
- > Click this link for Contextual Category Allocation Table – this shows how the sources were allocated contextual codes to place them into this system of ranking.
- > Click this link for Contextual Categories to PBH for top 6 themes
Through using a number of different groups of participants it was hoped that some kind of constructive comparisons might be made about perceptions and relative association or context between each subject group within the context of technology use in learning and teaching. Whilst sampling methods for this research project were minimal in their robustness (being non random or self selecting), there is still some attempt made to avoid bias by including a variety of types of participants, i.e. various job roles, or degree disciplines, so as to make some attempt at avoiding skewed results. It must be said however, that sample size as well as sampling techniques would need significant strengthening for any further research to be carried out to build on this pilot project.
The fluency and familiarity of technology tools informs many aspects of decision making and utilisation possibilities in relation to technology uptake, therefore to attempt to establish why and what stakeholders may think and do in relation to their technology habits was at the heart of the research for Research Group 1. To then be able to align this with the Rogers Diffusion of Innovations adopter categories was desirable, as Rogers model has been used in a number of other studies in this field (Sahan, 2006). To add some measurement of technology skills and experience into this model seemed logical, and could lead to further understanding in this field in relation to uptake of technology enhanced learning and teaching.
Rogers categories can be summed up below with a frequently used abridged set of terms for the types of people found in each category. More in depth descriptions are contained within Rogers own work (1995, 2003), or those which this research used, in Sahan’s ‘Detailed Review of Rogers Diffusion of Innovations Theory and Educational Technologies Studies based on Rogers Theory’ (2006).
- Innovator: Venturesome
- Early Adopters: Respectable
- Early Majority: Deliberate
- Late Majority: Skeptical
- Laggards: Traditional
A scale of proficiencies and efficacies was developed from data gathered in the question sets, and then used to allocate a ‘Rogers Diffusion of Innovations indicator’. Using their anonymised code name of R1, R2 etc, the profile of each participant was then placed into the Rogers scale of technology adopters, but with additional technical proficiency and efficacy awarenesses.
Our participants appear to be:
- 1 innovator (R1)
- 2 early adopters (R3 and R5)
- 4 early majority (R4, R6, R7, R8)
- 1 late majority (R2)
This was intended as an approximate indicator to add a technical aspect to the Rogers Adopter categories model.
Table 4: showing the RDI indicator allocations
Please refer to page 4 of the RG1 data analysis appendix for more details.
Questionnaire Set Results for top six themes
Questions were divided into short sets of around 8 questions each, and each set devoted to a specific topic. This helped to focus the thoughts of the participant, limit the participation time required and made it easier for respondents to continue with the research knowing it was not too time consuming. Six question sets were used in total. These are the main findings, with key points indicated. Please refer to the full questions sets, and the data analysis.
The data has been analysed for the top six themes only (themes with most occurrences in the literature), taking into account the time and depth limitations of this research project. However, this demonstrates the principle of how this type of data can be used in relation to literature themes and other related research group data.
Main key findings of each theme were collated with the theme’s perception as to it being a problem or a benefit. Overall they confirmed the placing of the theme, but there are some conflicts, and some challenging comparisons noticeable. Refer to page 3 of Research Group 1 Data for a full set of findings for the top 6 themes.
|Theme in Literature||Problem or Benefit
(Problem or Benefit)
|INSTITUTIONAL SUPPORT||Problems||Strongly perceived as posing problems|
|SOCIETAL CHANGES||Problems||Fairly perceived as posing problems|
|PEDAGOGY / LEARNING DESIGN||Benefits||Fairly perceived as offering benefits|
|STUDENT CENTRED LEARNING||Problems and Benefits||Fairly Ambivalent|
|LEARNING QUALITY||Problems and Benefits||Strongly ambivalent|
|CONVENIENCE / WORK LIFE BALANCE||Benefits||Fairly Ambivalent|
Table 4 showing problem or benefit setting of literature theme, with comparison of overall placing from question set data
The informal conversations (instigated by this project) which occurred through use of social media networks with an Academic group on LinkedIn and ResearchGate were both enlightening and worthwhile, demonstrating amply the abundance of thinking and opinion surrounding the topic. Though ResearchGate offered far less input, being a smaller and more specific network (therefore a smaller pool or participants to draw from), the HE LinkedIn group was a hive of activity for around 2 – 3 weeks, with nearly 40 comments made by experienced HE practitioners, both lecturers and support staff.
Results were categorised into most common themes mentioned and then assigned to the PBH hierarchy according to relevance. A table was created for the category to theme relationship, and then for theme to PBH allocation. Highlighted quotes of interest were used to generate suitable categories, then categorise and finally place into the research project themes.
The two most predominant response categories were ‘Top down/bottom up‘ and ‘Effectiveness (teaching & learning)‘. Top down/bottom up was then matched with the ‘Institutional Support’ literature theme, and Effectiveness was matched with ‘Learning Quality’. Within ‘Top down/bottom up’, the most common type of comment was quite negative, either with the respondent’s current experience, presumptions about reception or provision, or seeing problems with those areas for others, not necessarily themselves. For ‘Effectiveness’, there appeared to be significant polarisation in perception or confidence in technology having a positive effect (more or improved learning), as well as about methods by which such things are measured. Two camps seem to exist, those who embrace, sometimes with little proven evidence beyond their own experience, and those who would doubt any effectiveness at all.
Fig 1: Showing percentage of responses per category, RG2 (LinkedIn)
The chart in Fig 1 shows the close resemblance to percentage of occurrences for themes in the literature. The one area that differs is ‘staff motivation’, which may here be more evident in terms of self generated topics. In much research, the research is often dependent on questions instigated by the researcher. This demonstrates very well the additional value of using self generated feedback through social network channels.
Findings for Research group 3, the students, though limited in amount, were perhaps quite valuable as contrasting data to that gathered from staff. The amount of data tended to limit how much it might be analysed within the system, however, as anecdotal input, insight is increased.
Students are concerned about what they learn, and the potential of that learning to lead to greater knowledge. They are perhaps not so concerned with how they learn, though some consideration is given by them as to methods of assessment and communication, which was in itself quite enlightening. The other perhaps significant problem from a student perspective is the lack of technical skills of the lecturers, which came through quite strongly, even with the limited amount of data.
As the study process perhaps creates ‘a giver and receiver’ mentality, there was hesitancy in putting forward new ideas, and a sense of “you just get used to the way things are” (student quote), but there was also knowledge of the increased competitiveness of the university experience, as seen in another of the quotes.
Some quote highlights, which provide insight into TEL influencing factors from the student perspective were as follows:
- I’d like to see teaching move towards interactive technology, such as electronic whiteboards and the like. Granted the use of technology requires a great deal of investment, not just in equipment but in training and support, but at the end of the day it’s a necessity in order to compete with other providers.
- There’s always the stereotypical generation gap when it comes to the use of technology and I know not everyone is keen on advancements, but at the end of the day people need to realise that we’re not using chalkboards and abacuses anymore.
- There are facilities in Blackboard that are not utilised (course areas, for example) which could benefit feedback and interaction with students as a whole. As it stands we are not really encouraged to communicate by any means.
- We tried blogging for a piece of coursework and it didn’t work well as the other students just weren’t interested in participating.
- Quite frequently we have to shout out advice on how to open a link or something.
- More often than not they seem to know less as they struggle when they can’t set lecture slides up.
- It doesn’t matter if they are competent with technology or whatever as long they are able to keep their field alive inspiring their students to either follow the same path…
- There should be more opportunity for students to be technologically creative with their assignments across all faculties – I’m not sure why there is such a focus on essays which develop little in the way of communication and ICT skills (both of which are valuable in the workplace).
Fig 2: Showing percentage of responses per category, RG3
Problems and Benefits Hierarchy v1
Problems and Benefits Hierarchy v1
The first Problems and Benefits Hierarchy was mapped after the literature theme analysis had been completed, including the contextual categorisation. This showed that all of the top themes were ‘real’, with expert knowledge or research evidence being the predominant contexts of those occurrences. Pedagogy/Learning Design had the highest ‘real’ count, with 25 instances, and Convenience/Work Life Balance had the lowest, with 11 instances. The full details are available below.
Table 1: PBH v1
The first iteration of the Problems and Benefits Hierarchy was then iterated a further three times (see the Data Analysis Architecture diagram) to attempt additional validity and clarification for accuracy and evidence of theme rankings.
Problems and Benefits Hierarchy v2
The second iteration of the PBH was done after data had been established from RG1 in order to further validate the placing of the ‘problem’ or ‘beneift’ aspect of the theme, not its context. This reaffirmed the perception of whether a theme was a problem or a benefit, overall. Data gathered from questions asked to RG1 challenged initial impressions of theme placement, in that all themes were regraded as problems, though two were ambivalent. Pedagogy and Learning Design was also seen as a benefit in terms of reactions to positive potential of use of technology, and student centred learning, interpreted here in the broadest sense of putting the student at the centre of all aspects of university life, received inconclusive results as to potential or perceived expectations or benefits, which was somewhat unexpected.
Table 2: PBH v2
Problems and Benefits Hierarchy v3
To compile a meaningful third iteration which included qualitative data from RG2, some licence was taken to adapt analysis, though this was not in itself overly damaging to the value of this stage of data contribution. Response category data was allocated a set of context category values as would best allow, considering the data was more limited. However, more context was developed in this way, to add to the overall validity of the PBH rankings. In fact it is noticeable that contexts appeared similar to that of the literature, though this might be in part due to only one individual interpreting the data.
Table 3: PBH v3
Problems and Benefits Hierarchy v4
The final PBH was then developed to include data from RG3. Though this group also only provided limited data, some of which could not be included as was categorised into a theme which was not part of the top 6 themes, the data that was produced was allocated context category values, and added to the PBH. This did not make significant impact on the PBH, but with further research in this area, could be analysed first separately, and then placed into the PBH with more value.
Table 4: PBH v4
Overall Problems and Benefits Hierarchy
The final overview of the top 6 themes in terms of PBH contextual ranking shows that only one theme survives as a benefit, and that is classed as a problem too (Pedagogy and Learning Design). Both Learning Quality and Student Centred Learning become problems with ambivalence, meaning they are more weighted as problems but still have some presence as benefits, of less significance. Convenience /Work-Life Balance now shows as a problem.
Table 5: Problems and Benefits Hierarchy Ranking Overview