Skip to main content


ICC 650
Box 571014

37th & O St, N.W.
Washington, D.C. 20057

maps & directions

Phone: (202) 687.6400



Quality in Organic Data

Posted on

As social scientists increasingly encounter new data resources, especially those in the so-called “big data” realm, they’re finding a new challenge to identifying the proper quality framework to use.

For some years, much of empirical social science was guided by a framework of inference from a sample-based data set to a large well-defined population. The data and statistics derived from the data were evaluated through the lens of a “total survey error” framework, often presented in the chart below:

29 JUL 2015_Graph

Some of this framework focused on quality properties that caused biases (consistent, systematic error) between the population from which the sample was drawn and the sample generating the data. For example, if the survey data came from a web-based survey, the researcher has to attempt to measure the impact of missing persons with no web-access (this produced “coverage error” on the chart above). Would those without web access have given different answers to the survey questions than those with web-access?

We’re now inundated with statistics from Twitter and Facebook and other social media platforms, but few studies using such data ask the question whether the nonsubscribers to those platforms would be different on the statistics published.

Worrying about the biases in statistics due to missing observations, however, does not require a large renovation in the quality framework above used in surveys.

A more important difference between so-called organic data (e.g., from social media) and designed data (e.g., from surveys) is that the researcher controls the observations in designed data but does not in organic data. Much of the survey error framework acknowledges possible mismatches between the desired target of measurement (e.g., the status of being employed) and survey questions that are asked in the questionnaire (e.g., “Last week, did you do any work for pay”?)

With the new data resources available to researchers in the big data world, a different kind of measurement issue arises. The researcher is merely “harvesting” the “exhaust” of people as they live their lives. What’s in the exhaust is not controlled by the researcher. For example, what would motivate a tweet that says “I lost my job today.” What type of Twitter subscriber who did indeed lose their paying job would choose to tweet this? What type of Twittter subscriber who lost their job would choose not to send such a tweet? If a subscriber is unemployed, what is the probability that he or she would tweet evidence of that status repeatedly during their unemployment spell? Would a subscriber ever send such a tweet despite being employed for pay? Do people holding multiple jobs behave differently than those who hold only one job?

To construct a useful quality framework for such organic data, the researcher needs to tackle the question of why a person would choose to provide information on the platform. Understanding the motivation is key to knowing the signal to noise ratio in the data for a given phenomenon. The probability of creating such evidence must be known for both those who have the attribute and those who do not have the attribute.

This kind of quality feature of big data cannot be measured within the big-data set itself. Such biases aren’t corrected by having larger data sets; the errors stem from inherent mismatches between the target of measurement and the processes producing the data.

Further, we don’t have language for this type of data quality feature. Candidates might be “the propensity to report an attribute,” “the likelihood of signaling,” or “match between the attribute and the signaling.” None of these are pithy.

Great care will be required in the move from data that were designed for a specific analytic purpose to data harvested from digital traces naturally occurring. We need serious discipline about big-data quality.

Data, Trust, Verification

Posted on

Each day, we seem to be inundated with two types of media stories simultaneously — 1) how “big data” will usher in a world of heightened convenience and efficiency for all and 2) how relentless tracking of our personal information threatens our autonomy as human beings.

In prior decades, much of our collective understanding of how people felt about issues, what activities they pursue, and what knowledge they possess about key issues facing their lives, came from direct questioning them. The questions were components of sample surveys, through which a scientific sample of the full population was systematically measured and their answers statistically aggregated to describe the full population. The selected respondents to these surveys were given pledges that their answers would remain confidential to the survey organization, and only statistical aggregations would be constructed by combining their answers with many others.

One property of this prior world was that respondents were aware of which of their attributes would be known through the survey (i.e., only the questions answered by them). A second property is that their participation was voluntary, and the proposed uses of the data could be a factor in whether they chose to respond. A third property was that most institutions collecting survey data earned the trust of respondents that the pledges of confidentiality would be honored.

Over the decades, this protocol worked fairly well. There were very few violations of the confidentiality pledges. There was effective dissemination of information to the public to describe key features of their world — how well the government was perceived to be fulfilling their needs; how well-off the public was on basic attributes of income, educational attainment, and health status; how safe from crime different populations found themselves; and how well businesses were performing. That is, by sample persons giving up their privacy to provide data held confidential and used only for statistical purposes, the full society was informed about how well it was doing. Indeed, the data were designed to achieve this common good outcome.

Enter the Internet and unobtrusive data collection on persons, users, and members of services.

This new world produces data as auxiliary to other processes (traffic management, search algorithms, mobile phone location identification, social media communication, and credit card use). We, as individuals, use these services and in return to the personal benefits of the services, provide personal information to the service (this is generally authorized in the fine print of use agreements that most of us don’t read but quickly hit the “Agree” button).

These data are attractive to social scientists because they are fine-grained temporarily (some almost real-time), they are plentiful (trillions of observations versus thousands of survey respondents), and they track some behaviors that seem important to understanding how society is functioning. Will they become the equivalent of the ubiquitous survey data of the 20th century?

What’s new about this world is that the data weren’t designed to answer any particular economic or social question. Further, they are lean in number of attributes measured on each observation (i.e., we don’t know a lot about whoever initiated the data burst). Finally, they are not held by institutions whose mission is to extract information for common good purposes. Instead, they largely come from businesses that use the data to provide their services.

Most social scientists feel that this new data world has promise to unlock new insights about human thought and behavior. But it’s a different world — there is no defined infrastructure to coordinate the access to diverse data sources. It seems clear (to me, at least) that the winning society in the future will create a way to address privacy concerns of data access, private sector data holder concerns, and needs of researchers to combine diverse data to create more insights. This will require a new set of structures to assure privacy rights of individuals and verification that the data usage does indeed serve common good purposes. If the new world does not combine these new data sources for common good purposes, we will all take a step backward.

Intent of the Communicator, Comprehension by the Listener

Posted on

The University of California (UC) has issued some guidance attempting to ameliorate the bad feelings generated among students of color and other groups when various statements or actions occur by those outside these groups. These kinds of behaviors were often mentioned in the Georgetown students’ Twitter campaign (see #BBGU, #BAGU, #BLGU, and others). The guidance has generated much controversy. The guidance has a bit of a checklist of what types of statements risk giving offense.

Some of the speech acts that are highlighted seem directly to involve assumptions by the speaker: “You are a credit to your race” or “You’re a girl, you don’t have to be good at math” or “You people…” The person to whom this is directed, the listener, has to process what is intended by the communication. On the surface, it’s easy to see why the speaker may be viewed as making judgments based on generalizations that do not take into account the unique attributes of the listener. Alternatively, they are indirectly revealing assumptions about entire groups of people, some falsehood about homogeneity within a group. Of course, what the speaker intended is rarely articulated. It’s also easy to imagine what the listener thinks about the statements. “I don’t want to be defined by a single observable attribute.” “They view my group entirely differently than I view my group.”

Other statements are less directly targeting the subgroup: “Why are you so quiet? We want to know what you think. Be more verbal.” and could be said to many students. The intent of the speaker remains unclear, but the UC document notes that when the listener is of a culture in which verbal interaction is governed by norms different from American ones, the communication may not achieve the intended outcome. The document notes the possibility that what is heard is a devaluing of the culture of origin of the listener.

The UC effort is consistent with growing evidence that we ourselves are sometimes not privy to our own assumptions. We act and speak often with implicit assumptions. Our effortful cognitions yield different outcomes than our automatic actions.

The controversy that ensued over the UC guidance forces attention to one of growing concern — when does speech become harmful? What uncomfortable statements should be removed from day-to-day speech? When is the perceived intent of a speaker itself based on unfounded assumptions? When should the listener think more deeply about the intent of the speaker? How can we learn to speak effectively to those who are members of groups outside of our own? How can we see the world as they see it? How can we conquer our own fear of giving offense, in order to create real communication across group barriers?

With the events of this past year in the US, it seems clear that we collectively have work to do. The understanding across groups that we lack needs an honest dialogue, reflection, and more dialogue. This will require ongoing work. It seems to be a wonderful opportunity for US universities to lead this effort, and Georgetown’s excellence in inter-group dialogue gives it strengths that should serve it well in these tasks.

Team Learning for the Fun of it

Posted on

Over the past few weeks, I’ve found myself in various national meetings discussing the future of higher education and new pedagogical vehicles that seem to fit that future. It’s led me to reflect on personal experiences that form my deepest and warmest memories of organized learning.

The first was a course on Joyce in the English department in my sophomore year. The instructor was a famous senior professor. He entered the classroom with his young dog, whom he instructed to sit below the instructor’s desk. (I immediately thought that was cool and am still amazed at how disciplined the dog was throughout the semester.) I’ll never forget his first words in the class: “I’ve never taught James Joyce before, but I’ve wanted to do so over the years. I’m not sure I know what I’m doing; you and I will learn together about his writings.” While this was an impressive unveiling of his vulnerabilities, it was just the beginning of his setting up an environment giving us permission to reveal our own insights into the writing. The readings were voluminous. In addition to Joyce’s major works, we read multiple books of literary criticism of the writings. The classes were spent sharing our interpretations of the writing, with an openness for alternative meanings. By never openly teaching us, the professor allowed us to learn the layers of meaning of the words, sentences, paragraphs, and actions in the texts. We felt like we were all working together as a team; we were both students and teachers. The lessons of reading and re-reading, letting the text soak in my consciousness, were never forgotten. I kept my notes and course materials for the class for over 20 years.

Another of the most heartwarming experiences of my life was, in retrospect, a near-perfect union of the university goals of formation of students and original scholarship.

The story begins with the observation that multiple courses in my university were tackling the same topic, but from very different perspectives. The courses were offered in different disciplines. The particular issue was the impact on statistics of failure to measure all members of a sample within a survey context. One course began with the existence of a data set, already assembled, but subject to the missing data problem. Another dealt with the cognitive and behavioral underpinnings explaining why sample persons would provide data (or not provide data) to a sample survey. I taught one; a colleague taught the other. There was no overlap in the readings of the courses.

I teamed with my colleague to build a new course from scratch. The faculty taught the course for free. The course wasn’t required for any program. The other instructor and I began this course by presenting the perspective of our disciplines toward the material. Then, the class and we together attempted to look for differences and similarities in the approaches. We moved to constructing a synthesis of the approaches. Toward the end of the class, we decided that we were forming an approach that really needed more work. The semester ended at the point that we formulated a research design to test out new ideas that arose during the term. In a way the course produced new knowledge, yet untested with rigorous research, but we knew what analyses on what existing data needed to be mounted.

As the semester ended, we were surprised — the students asked whether we could continue meeting, with or without a formal course. We decided to plow ahead; everybody became part of a research project. We divided up the work; we became a project team. By the end of a few months, we had the ingredients of what became a journal article and the vision of several others produced later. It was great fun; the ratio of learning per time unit was unusually high.

To me, there are similarities in these two experiences. Both of them stimulated active learning because there was a sense that new discoveries were possible. Both of them were fresh, new, and filled with untested content. Both of them generated a sense of a team of peers attempting to understand in new ways. Both of them permitted all involved, both students and faculty, to become learners and teachers. Both of them involved a lot of work but even more fun.

I marvel at how these experiences resemble some of the project-based and research-based learning protocols being created as part of Designing the Future(s) of Georgetown. If Georgetown could increase the frequency of these kinds of courses, I’m convinced our graduates may have more experiences that they remember years later. I also believe our faculty would have more fun teaching.

The Evolution of Professions

Posted on

There’s rapid change occurring in traditional professions. The common theme appears to be an evolution from self-directed work to activities in support of the mission of a larger organization.

Physicians, once the epitome of the self-employed, are increasingly employees of large organizations, members of big teams of co-workers. Lawyers, another profession where “hanging out a shingle” was the metaphor to launching a career, now increasingly find themselves within large organizations whose mission has legal guidance only as an auxiliary function. Many architects work within units of institutions that require design skills, but only because they need to house large groups of workers performing activities that have nothing to do with architecture. These are examples of traditional professions.

The morphing also seems to apply to products of graduate education in more traditional disciplines. A recent report of the National Science Foundation notes that many STEM trained graduates find themselves working in fields without major STEM focus. Another report on physics PhD’s in nonacademic settings probes levels of salary and job satisfaction. The Modern Language Association 2014 report argues for more interdisciplinary content to PhD programs in language and literature. In contrast to the decline of self-employment among physicians, lawyers, and architects, this attention appears to be associated with the decline in academic job markets, related to demographic changes in the university-age U.S. population and declining government support for higher education. Increasingly, PhD’s are working in large nonacademic organizations.

In short, something’s afoot on several fronts regarding the post-baccalaureate higher education graduates. What are the implications for educational institutions that launch such new professionals into the world?

It’s seems fair to say that the traditional role of many graduate professional and PhD programs was to provide the graduate very deep and broad understanding of a well-defined body of knowledge. Any curricular feature that strayed outside that well-defined body of knowledge was generally meant to prepare for a single dominant occupational class of the profession. For example, law students were given training in moot court settings and in clinics with individual clients (often disadvantaged persons). Medical students were given clinical experiences through rotations in various specialties within a direct health care service or in private practice. Architects served long apprenticeships within an architectural practice. PhD students were given teaching experience. In short, the programs offered practical experience in activities central to the execution of the dominant occupation of the profession.

In contrast, few PhD and professional graduate programs tended to provide education in the knowledge needed to be effective leaders in large organizations. However, increasingly the graduates of these advanced education programs find themselves working within large organizations. In these organizations, they are not self-directed scholars. They are team members working to achieve the mission of the organization, a mission sometimes only tangentially related to their advanced education. They feel the dissonance of self-identity to their profession versus allegiance to the organization’s success. They work with others completely unschooled in their field. They find themselves leading others, supervising others, and motivating others. They find themselves confronted with budget constraints, making tradeoff decisions among alternative goals (some completely out of their professional domain), forecasting production, analyzing performance, and dealing with personnel problems.

It seems clear that, for the most part, universities have been slow to recognize the mismatch between how they are educating these advanced professionals and academics, on one hand, and what these graduates will need to know to achieve success. There are some programs, both at Georgetown and other universities, that attempt to give PhD and professional students more interdisciplinary education to help them attain leadership positions within larger organizations. These have great merit in serving the changing job markets of advanced degree holders.

However, one of the more difficult challenges of advanced programs is to educate for the new career realities of a field instead of replicating the education of the current leaders of a field. We need to ask ourselves whether we are training students for careers we experienced or training them for the careers they will experience. For the benefit of our future graduates, we need to attend to these issues.

Students as Producers Versus Consumers

Posted on

Hunter Rawlings, the president of the Association of American Universities, the consortium of the large research universities in the country, recently wrote a thoughtful essay challenging various beliefs about the key purposes of universities.

As more and more commentators are noting, assessing the value of a university based only on lifetime earnings (or even more radically, the income of the first job post graduation) misses many of the components of the experience.

However, Rawlings makes another point — thinking of university degrees as something to purchase, as a consumer product is also a dangerous misunderstanding. Students maximize the value of their higher education by maximizing the effectiveness of their studying. The more the students give of their own time, the higher the value of the purchased experience.

As we are discovering in the Designing the Future(s) activities, much learning of students takes place out of the classroom, in on-the-job experiences connected to their educational programs and in research-based work. The vast majority of these activities are situations in which the students are actively teaching themselves. That is, the students are investing their own time to achieve the benefits of the education.

In some sense, students are not consumers purchasing a good, they are producers of their own education guided by faculty mentors. They are not buying teaching; they are subscribing to a set of experiences that allows them to discover their talents and interests.

Rather than thinking of acquiring higher education as the purchase of shares of stock, in which a chief purpose is a passive return on investment, maybe it’s better to think of higher education as a fitness center where the chief purpose of the membership is better health status. The membership fee gives one access to exercise facilities and machines, but the health status benefits arise only if the member optimizes their use. The “return on investment” is fully in the hands of the member.

Good health through exercise can be achieved in many different ways. Some people thrive in group activities, using others to heighten motivation. Some emphasize cardiovascular health through aerobic exercise. Some lift weights. Choosing the right fitness center, like choosing the right university, is a matter of matching aspirations, self-knowledge, and the assets of the organization. But, just like a fitness center membership, students get their “return on investment” when they take advantage actively of all the resources of the organization. When students take charge of their education, direct it, invest deeply through their own time, only then do they gain the benefits of the higher education institution.

When students come to realize that they are the producers of their education, not consumers, then, higher education productivity is maximized.

Evolving a Supportive Environment for Interdisciplinarity

Posted on

I’ve written much in the past about alignment between universities and the wider world (see Interdisciplinary, Collaborative). Many of the problems in the world will not be solved with the knowledge from single disciplines, and, hence, universities need to make sure that they build environments in which faculty and students can easily combine work in multiple disciplines.

At Georgetown, we’re moving in that direction. We’ve reorganized the Graduate School to build new interdisciplinary programs. They will be built around coalitions of faculty in diverse units throughout the university. This is proceeding, and faculty who are working on similar issues from different disciplinary perspectives are coalescing around working together on such programs.

In designing these programs, a key goal will be liberating faculty who wish to work on such interdisciplinary issues, but also permitting them to maintain their citizenship in the current home departments. Such joint citizenship is not for everyone, nor should it be. It should, however, be available for those who thirst for such scholarship and instruction.

It must also be done in a way not to harm the existing departments’ delivery of their curricula. This means that the talent of faculty available to units must fit the needs of the enrollments in the current curricula.

To prepare for this new environment, we have altered the nature of the joint appointment process, setting up merit review and promotion procedures to reflect appropriately the contributions of the faculty to the interdisciplinary agenda (see Faculty Policies for joint appointment policy).

Over the coming weeks and months, we’ll talk more about joint appointments and how we can build an environment that supports them and those who hold them.

A critical component of this is the promotion step, where we must assure that those evaluating the performance of the faculty member understand the interaction of multiple fields and how to assess quality for that intersection. If the interdisciplinary field, for example, draws on two disciplines, then three sets of expertise are needed — those from each of the fields and those from the interdisciplinary intersection. Peer review must be based on those who are experts in the interdisciplinary area.

Disciplines and schools vary in their support of interdisciplinary work. Thus, incentives must be tailored to the cultures of the various units. Over the coming weeks and months, we’ll engage in the necessary dialogue to fashion such incentives.

Georgetown has consistently tackled the greatest issues facing humankind. The world needs Georgetown even more in the future, and facilitating joint appointments for those tackling the world’s most important problems is a wonderful way to address those needs.

Labor Economics, Exponential Organizations, Mentors, and Well-Being

Posted on

Many of us find ourselves reading multiple unrelated works at the same time and combining disparate works into a single observation. That’s been happening to me over the past few days.

First, I was looking at some older research in labor economics on the impact of technology on jobs. There has been large destruction of jobs in some sectors due to technological change. However, a now well-established finding is that (thus far) there are several job categories that appear less affected by technology. They include work that requires complicated interpersonal, cognitive, and physical skills — policemen, dentists, editors, artists, counselors, and surgeons. None of us know, of course, whether these occupations are merely not yet affected by technological disruptions or whether there are essential parts of the occupations that can never be replaced by machines. Technological progress appears to be on an unceasing advance. But for the time being, the rapid changes that have taken place in communication, financial transactions, retail commerce, manufacturing, etc., seem to have evaded these occupations. The same is currently true of academic faculty in higher education. And so, I think about why faculty activities have been so robust to technological changes.

Second, I’m reading another work right now that coined the phrase “exponential organization,” one whose growth is not linear but multiplicative. The most obvious examples are Uber, Google, etc. A common factor in these organizations is that they are much less dependent on human labor for their product or service. Most organizations, indeed, depend on information and algorithms as key components of their operations (Uber uses ratings of drivers and customers as a key tool; Google organizes information to be useful to unique user requests). Such organizations lie in stark contrast to those in which human labor produces the service or outcome — a craftsman produces a table in one week; two craftsmen produce two tables in one week. Labeling such enterprises as linear organizations, the authors argue that they are doomed to slow growth (and, they assert, limited nimbleness) because of the requisite trials and tribulations of human collaboration and authority structures. All of this is interesting because from one perspective, a university is the ultimate information-based organization.

Third, as part of our attempts to measure outcomes of our degree programs, I’ve been reading the results of various surveys of graduates. One attempted to measure engagement in one’s work and one’s general “well-being” among university graduates. The study found that graduates’ self-assessed well-being was twice as high among those who had had a faculty mentor who encouraged them to pursue their own goals, one who had made them excited about learning, and one who had cared about them as a person. These are powerful results, linked to a set of interpersonal relationships between individual faculty and students. Similar results apply to the outcome of whether the graduate is deeply engaged in their current occupation. Clearly, mere transmission of information from the minds of faculty to the minds of students does not alone achieve these positive outcomes.

When I put these disparate pieces of new reading together, I can’t help linking them to the last 18 months of global experience with online learning. With great hubris, we saw the launch of MOOCs, which self-labeled as a “fundamental disruption” to universities. Their laudable promise was making higher education accessible to all with internet access. It was a wonderful moment of exuberance. The idea had all the ingredients of an exponential organization; at least, that’s what we all were thinking at the moment. But they did not prove to launch exponential organizations. The audience was disproportionately not those who needed education, but those who already had an education — and already had an inner-thirst for learning. They were disproportionately those who already had learned how to learn and wanted more. There are far fewer such people in the world than those who have never experienced higher education.

Reflecting back on that moment and on my current three sets of reading, one hypothesis emerges. While much of the value of universities is related to their rich store of information within the minds of the faculty distributed to students, there is much more going on. The self-reports of the more successful in life (in both quality and engagement) suggest that deep interpersonal ties with faculty is part of the magic sauce of US higher education.

So an interesting question is how can we enhance the deep interpersonal mentoring that so enriches the intellect and character of graduates. Can technology actually free faculty to have more of that rich contact? Could the portion of faculty work that doesn’t offer such interpersonal opportunities be reduced through online tools? Can technology be used to transform the way students learn what is traditionally communicated in lectures? Can lectures then be replaced by small group work in direct contact with mentors? Could faculty research indeed be the credit-bearing project work that simultaneously addresses faculty scholarship and closer mentoring? In essence, which of the three missions of the university (formation of students’ characters, faculty scholarship, and service to the common good) can be improved through new technologies? Which can’t?

In essence, is there a portion of the mission of universities that could be organized in ways that “exponential organizations” are shaped, freeing up other portions of the mission to be enhanced? Those other portions might be inevitably “linear” in their growth, but with precious value to the outcome in students’ lives.

Learning by Seeing the Real Thing

Posted on

Yesterday, I attended the official opening of the Booth Family Center for Special Collections at the Lauinger Library. The newly constructed space, made possible by the generosity of the Booth Family and others, has all the possibilities of transforming Georgetown research and education. The Center is the new home for rare books, art works, and manuscripts acquired by the University over the centuries. In addition to being state of the art for the conservation of materials, it looks “cool,” with glass walls and high-tech features.

There is much talk these days about experiential learning, project-based learning, and learning-by-doing. Indeed, much of the work of the Georgetown Red House incubator, part of the Designing the Future(s) initiative, focuses on how we can move beyond the reading + lecture + exercises/paper + exam format of university courses. Complicated concepts communicated orally or in written form don’t often stick in the minds of the hearer. Research in the science of learning repeatedly shows the value of applying newly-found knowledge in real-life settings.

The same logic applies to the potential of the Special Collection facility. For students interested in American literature, for example, we have the manuscript, in Samuel Clemens’ own handwriting of The Adventures of Tom Sawyer, as it was sent to the printer prior to publication. Imagine you’re a student trying to delve deeper into what Samuel Clemens was intending to communicate in his writing. What did it mean that he scratched out one phrase in the editing of the book and replaced it with another phrase? As one passes through all of the edits, is there any theme that he was consistently trying to elaborate? Which chapters were subject to more late-stage editing than others? Was the writing regarding some characters in the book (Aunt Polly, Injun Joe, Huck, Tom, etc.) subject to more edits than that other characters?

None of these questions can be answered by reading the printed book. No digital image of the manuscript can convey the emotion that comes from touching the same piece of paper that Samuel Clemens handled. Examining the object of study itself stimulates unanswered questions and offers clues of answers of questions with unparalleled power. Even more important, it’s fun.

None of this kind of learning and scholarship is possible without the conservation of the objects of study. Objects of art, old manuscripts, and old book volumes must be protected. Temperature and humidity can affect their life. Uncontrolled access by those untrained in how to protect old objects can ruin the objects for future generations.

What makes the Booth Family Center so wonderful for Georgetown is that in addition to climate-controlled environments to protect the various objects, the facility contains a high-tech classroom that permits group examination of the objects. The classroom will be the place where students learn to examine the objects in a way that assures their long life and access to researchers of future generations. The facility can be the site where students and faculty interact in original scholarship, with faculty communicating the deep inquiry and intense observation of such objects necessary for insight.

For the areas of inquiry whose raw materials are these objects, this is experiential learning at its best. The students will be using the “real thing” as part of their work; real research skills that can apply to future scholarship will be communicated. To my eyes, the new facility is so attractive as a space that I expect it will breed a new generation of scholars who find it fun to work inside it.

Assessing Scholarship Across Disciplines

Posted on

One of the challenges of a modern university is assessing the quality of the scholarship of the faculty. It’s generally agreed that quality of the product of a scholar is a function of the novelty or uniqueness of the work and the impact it has on later research in the field.

There are various complications that arise in assessing the scholarship of faculty. Many of them stem from different intellectual cultures across the disciplines.

Some fields use book-length products as the coin of the realm; others use journal articles of 10-20 printed pages as the basic dissemination device. (Still others use created objects or performances.) As one might imagine, these differences are related to quite different volumes of products across fields. Books take longer to produce on average than an article; so, at any given moment the odds that a faculty member has just produced a book are lower than a faculty member in another discipline producing an article. This complicates the evaluation of a faculty member’s product. In book disciplines, for example, what is to be inferred about the lack of publication for a few years?

Even within the disciplines mainly producing journal articles, there are complicated disciplinary differences. One impact measure commonly used is the number of times an article is cited by other authors in later work. However, fields appear to use citations to earlier work at different frequencies. For example, the average number of citations among economists is 28, but among psychologists is 127. What are bases of those differences? There are a lot of complicated reasons why fields might differ in the degree of citations. For example, much of psychology is an experimental research field; mounting of lab experiments is relatively cheap and fast. The inference from the experiments is that certain causal patterns can occur under the experimental conditions. The cumulative nature of the field depends on the replication and generalizability of the causal discoveries. Hence, the need to cite other related work to build the case that the experimental conditions are generalizable.

Fields also differ on the use of team-based scholarship. Some promote the culture of the lone scholar, with sole authorship dominant. Others do their work in teams, with a division of labor that often represents different specialty knowledge or phases of the research project (design, conduct, and evaluation/analysis). Some fields use very large teams. One of my favorite examples is a physics article in which there were so many authors that the first page of the journal was filled only by the author list! The standards of what constitutes sufficient contribution to merit a co-authorship also varies. Some faculty generously include student assistants as co-authors; others, do not. So, care must be given in assessing the contributions of an individual faculty member to team-based research.

Fields also vary in how contributions are reflected in the order of multiple authors. Many economics articles use the rule that the authors are listed alphabetically; in other fields, some two-person ongoing collaborations routinely rotate the order of authors over repeated articles. Some fields rigidly order by importance of the contribution of the author. Other fields give special meaning to the last person in the author list. Hence, without knowing the cultural norm that dictated the author order, it’s difficult to judge the contribution of a given author.

Finally, fields that are emerging as new knowledge domains have special problems. When they are combinations of existing fields, the scholar faces difficulties getting his/her work “noticed.” For example, in interdisciplinary work combining two fields, the flagship journals of the two different disciplines will rarely highly value such work, viewing it as a mere application of existing knowledge in the field. Only when new journals emerge, fully representing the new interdisciplinary space, can a proper evaluation framework be established. Even then, the readership of the new journals is typically lower than those of established fields.

So, for all the reasons above, evaluating the quality and impact of scholarship doesn’t easily lend itself to simple counting. While within-field quantitative comparisons are often worth making, cross-field comparisons based on counting are error-prone. Because of the diverse cultures of different fields, deep knowledge within the field is required to know what are high quality contributions and what are the best ways to measure the impact of those contributions.

Office of the ProvostBox 571014 650 ICC37th and O Streets, N.W., Washington D.C. 20057Phone: (202) 687.6400Fax: (202)

Connect with us via: