Fix the mistake: Information and Data Science

Data Collection and Organization・Levels of Measurement

Data has become a resource that drives society, to the extent that it is called “the oil of the 21st century.” However, data that has merely been collected cannot be used as it is. It creates value only when it is collected according to the purpose of analysis, its type is determined, and it is carefully organized. In this lesson, you will learn how to conduct data analysis, the types of data (levels of measurement), and preprocessing (data cleansing).

The Data Analysis Cycle — PPDAC

Data analysis does not begin with a sudden idea; it proceeds through the following five-stage cycle. It is called the PPDAC cycle, using the first letters of the stages.

① Problem: Decide the problem to solve or the question to clarify.
② Plan: Plan what data to collect and how to collect it.
③ Data: Collect and organize data according to the plan.
④ Analysis: Analyze the data using statistics and graphs.
⑤ Conclusion: Summarize the answer to the question and, if necessary, return to a new question.

Ways to Collect Data

Data can be collected independently through methods such as questionnaire surveys and experiments. Other methods include automatic collection using sensors (such as temperature and location information), using open data published by governments and local authorities (such as e-Stat), and obtaining data through a Web API. When using other people’s data, pay attention to the source, terms of use (license), and handling of personal information.

Structured and Unstructured Data

Data that is neatly organized into rows and columns, like a spreadsheet, is called structured data. In contrast, data that is not in table form, such as text, images, audio, and video, is called unstructured data. Most data in the world is unstructured, and its use is rapidly expanding with the development of machine learning.

Levels of Measurement — Understanding the “Ruler” of Data

Data is first broadly divided into qualitative data, which cannot be represented numerically, and quantitative data, which can be represented numerically. It is then classified into the following four levels of measurement. The calculations and graphs that are allowed differ depending on the level.

・Nominal scale (qualitative): Categories used only for distinction. Examples: blood type, prefecture of origin, sex. Only equality or inequality can be compared.
・Ordinal scale (qualitative): Order has meaning, but intervals do not. Examples: satisfaction (satisfied・neutral・dissatisfied), five-level grade evaluation. Size can be compared, but “differences” cannot be calculated.
・Interval scale (quantitative): The intervals between scale marks are equal, but 0 does not mean “nothing.” Examples: Celsius temperature, calendar years. Differences have meaning, but ratios such as “twice” do not (20℃ is not twice as hot as 10℃).
・Ratio scale (quantitative): 0 means “nothing,” and ratios have meaning. Examples: height, weight, amount of money, length of time.

Example Let us classify a “marathon race record table” by level of measurement.
・Bib number → nominal scale, used only to distinguish runners (the size of the number has no meaning)
・Rank (1st place・2nd place・3rd place…) → ordinal scale (the “difference” between 1st and 2nd may not be the same as that between 2nd and 3rd)
・Temperature on the day (℃) → interval scale (0℃ does not mean “no temperature”)
・Finish time (minutes) → ratio scale (0 minutes is the origin, and “twice as much time was required” has meaning)
Even when the “numbers” are the same, their meanings are completely different. It becomes clear that calculating the average bib number is meaningless.

Data Cleansing — Preparation Before Analysis

Newly collected raw data often contains problems. The process of organizing it is called data cleansing (preprocessing), and it is an important stage that takes up much of the time spent on analysis.

・Missing values: Data in which a value has not been entered. Possible actions include excluding the row or filling it in with the mean value.
・Outliers: Values that are extremely far from the others. Correct or exclude them if they are input errors, but do not delete them carelessly if they are genuine values.
・Inconsistent notation: Different representations with the same meaning, such as “Tokyo,” “Tokyo Metropolis,” and “Tokyo.” Aggregation will be inconsistent unless they are standardized.
・Duplicate data: The same record registered twice. Remove duplicates before aggregation.

Key Points
  • Analysis proceeds through the PPDAC cycle (Problem→Plan→Data→Analysis→Conclusion).
  • Data is divided into qualitative (nominal・ordinal) and quantitative (interval・ratio) data.
  • Nominal = distinction only, ordinal = order only, interval = differences have meaning (0 is arbitrary), ratio = ratios have meaning (0 is the origin).
  • The calculations allowed differ according to the level of measurement. An average can be calculated for quantitative data.
  • Data cleansing, which organizes missing values, outliers, and inconsistent notation, is the foundation of analysis.

Practice Problems

  1. State the level of measurement for each of the following data. ① Favorite school subject ② Five-level questionnaire rating ③ Temperature in Celsius ④ Body weight
  2. A blank (missing value) was found in a questionnaire response. Give two possible ways to deal with it.
  3. Are text and photographs posted on social media structured data or unstructured data?

Answers・Explanations

  1. Answer: ① Nominal scale ② Ordinal scale ③ Interval scale ④ Ratio scale
    Explanation: ① is only a classification. ② has an order, but there is no guarantee that the “difference between 5 and 4” equals the “difference between 2 and 1.” ③ 0℃ does not mean “no temperature.” ④ 0kg is the origin, and ratios have meaning.
  2. Answer: ① Exclude rows (responses) with missing values from the analysis. ② Fill them in with the mean, median, or another value from the remaining data.
    Explanation: Both methods have disadvantages (exclusion reduces the amount of data, while imputation may differ from the true value). It is important to consider why the missing value occurred before choosing.
  3. Answer: Unstructured data
    Explanation: Text, images, audio, and video are not in a table format of rows and columns, so they are unstructured data. Items such as “number of likes” and “posting date” can be treated as structured data.

Information (Information II): related lessons