Finding hypotheses with AI
To further analyse the issues described in ‘Problem-solving methods and thinking and experimental design’, it is necessary to find hypotheses. The following steps can be considered for finding hypotheses using AI:
1. data collection and pre-processing: finding hypotheses requires a wealth of data relevant to the problem in question, which is collected from a variety of sources, including sensors, user logs and survey results. Furthermore, if the data is used as is, there are problems with noise and missing values as described in ‘Noise removal, data cleansing and missing value interpolation in machine learning’, so cleaning, normalisation and feature engineering as described in ‘Various feature engineering methods and their implementation in python’ are used.
2. Exploratory Data Analysis (EDA): visualising data distributions and correlations to look for patterns and to see events and trends that provide clues to hypotheses. For example, it can be used to detect trends, differences between classes and outliers in time-series data, as described in ‘Time-series data analysis’, to construct hypotheses or to discover relationships between features using visualisation techniques such as scatter plots, histograms and correlation matrices.
3. use of machine learning models: train machine learning models and analyse the results to find new hypotheses. For example, models such as the decision tree described in ‘Overview of decision trees, applications and implementation examples’ and the random forest described in ‘Overview of random forest ranking, algorithms and implementation examples’ etc. can easily show which variables are influencing the target and thus generate hypotheses based on important variables and correlations. This allows for the generation of hypotheses based on important variables and correlations. Also, if a price forecasting model finds that a particular feature has a significant impact on forecast accuracy, it is possible to consider hypothesising that the feature may indicate some causal relationship.
Furthermore, by using unsupervised learning such as clustering as described in ‘Clustering in R – k-means’ and dimensionality reduction (e.g. PCA) as described in ‘On Principle Component Analysis (PCA)’, etc, Finding hidden patterns in the data and developing them as hypotheses, or finding groups of similar data and deriving hypotheses such as ‘this group is influenced by a specific factor’ based on the features common to each group.
4. hypothesis generation by reinforcement learning: by utilising reinforcement learning (Reinforcement Learning), as described in ‘Theories and algorithms of various reinforcement learning techniques and their implementation in python’, a hypothesis can be found through the process of the agent repeatedly obtaining rewards through trial and error. For example, the system observes what results are obtained when the agent takes a particular action and, based on the results, formulates a hypothesis that ‘this action causes a particular result’.
5. automating hypothesis testing with AI: For the hypotheses found, the causal inference techniques described in ‘Overview and implementation of causal inference and causal search techniques’ can be used to explore causal relationships between variables, thereby automatically testing hypotheses such as ‘whether A affects B’. Approaches to this include randomised trials and the use of causal networks (e.g. Bayesian networks).
Alternatively, hypotheses are tested in a virtual environment by simulating data using generative models (e.g. GANs and VAEs described in ‘Overview of GANs and various applications and implementation examples’). This allows the AI to mimic scenarios that would not occur in reality and to formulate hypotheses based on the results.
6. hypothesis generation using natural language processing (NLP): in addition, the natural language processing techniques described in ‘Natural Language Processing Techniques’ are used to automatically analyse large volumes of literature and research articles to discover existing knowledge and new hypotheses. For example, topic modelling and text mining, as described in ‘Overview and various implementations of topic models’, can extract relevant research areas and hidden knowledge, and find past hypotheses or research gaps on a particular topic from a set of research papers, from which new hypotheses can be generated.
7. hypothesis generation using AI-based anomaly detection: anomaly detection (described in ‘Anomaly detection and change detection techniques’) is good at finding data that deviates from normal patterns, and from this anomaly leads to some hypothesis. For example, if anomalous behaviour is detected in a data set, exploring the factors behind it can lead to new discoveries and hypotheses.
8. introducing an automatic hypothesis generation system: by using AutoML (automatic machine learning), as described in ‘Overview of automatic machine learning (AutoML), algorithms and various implementations’, AI automatically performs feature selection and model building, and automatically generates a hypothesis based on the results. This makes it possible to discover meaningful patterns in the data and efficiently proceed to hypothesis testing.
These AI-based methods for finding hypotheses make it possible to find new findings and hypotheses that would be overlooked by conventional methods.
Configuration of the automatic hypothesis generation system
Consider the configuration of a specific automatic hypothesis generation system. As a system configuration, we consider one that supports the process of automatically generating and testing hypotheses from data using AI technology. They consist of the following elements.
1. data collection module
- Role: collects and feeds the system with the data required for hypothesis generation, collecting real-time and historical data from various data sources (e.g. sensors, user data, literature databases).
- Functions:
– Data collection API
– Scraping tool
– Ingestion of real-time streaming data
– Database connection - Supported technologies:
– Databases (SQL, NoSQL)
– Web API, cloud data storage
2. data pre-processing module
- Role: cleans the collected data, prepares it for optimal hypothesis generation, completes missing values, normalises data, converts data formats, etc.
- Functions:
– Data cleaning (noise removal, missing value processing)
– Data normalisation and standardisation
– Feature engineering (extraction and transformation of important features)
– Outlier detection and processing - Supported technologies:
– Pandas, NumPy, Scikit-learn (Python library)
– Spark, Hadoop (large-scale data processing)
3. exploratory data analysis (EDA) module
- Role: to search for patterns and correlations in the data from which hypotheses are derived through data visualisation and statistical analysis.
- Functions:
– Data visualisation (histograms, scatter plots, correlation matrices, etc.)
– Correlation analysis
– Trend and anomaly detection - Supported technologies:
– Matplotlib, Seaborn, Plotly (data visualisation)
– Tableau, Power BI (visual analysis tools)
4. machine learning module
- Role: train and apply machine learning models to derive hypotheses from data. Models used here include supervised and unsupervised learning and reinforcement learning.
- Functions:
– Training and evaluation of models.
– Interpretation of models (extraction of important features)
– Clustering and anomaly detection
– Hypothesis generation based on model results. - Supported technologies:
– Scikit-learn, TensorFlow, PyTorch (machine learning and deep learning frameworks)
– AutoML (Google Cloud AutoML, H2O.ai)
5. natural language processing (NLP) module
- Role: to analyse literature and research articles to extract knowledge for hypothesis generation and to find research and trends to support hypotheses.
- Functions:
– Processing of articles and textual data.
– Topic modelling and keyword extraction
– Detection of research gaps
– Automatic generation of candidate hypotheses from text - Supported technologies:
– SpaCy, NLTK, Hugging Face (NLP library)
– GPT model (text generation)
6. causal inference module
- Role: explore causal relationships to test hypotheses, assess whether they support a hypothesis, AI finds causal relationships between data and validates hypotheses
- Functions:
– Building causal inference models (e.g. Bayesian networks, causal diagrams).
– Causal analysis of randomised trials and observational data
– Visualisation and interpretation of causal relationships - Supported technologies:
– DoWhy, CausalNex (causal inference library)
7. the Generative Models module
- Role: utilises generative models (e.g. GAN, VAE) to simulate hypotheses in a virtual environment, test hypotheses that are unlikely to occur in reality and generate new hypotheses based on simulation results.
- Functions:
– Generation of simulation data
– Evaluation of simulation results and improvement of hypotheses - Supported technologies:
– GAN (generative models), VAE (variational autoencoder)
8. hypothesis evaluation and validation module
- Role: quantitative evaluation of hypotheses and validation of their validity on the basis of actual data and additional experimental data; evaluation of hypotheses uses statistical tests and accuracy metrics.
- Functions:
– Hypothesis testing (statistical tests, accuracy assessment)
– Model performance evaluation (A/B testing, cross-validation)
– Ranking and reporting of hypotheses - Supported technologies:
– Scikit-learn, Statsmodels (statistical tests)
– MLflow (model evaluation and tracking)
9. hypothesis management and tracking module
- Role: to manage generated hypotheses, associated data and experimental results, and to store historical hypotheses and results. Tracking of hypotheses is important for long-term discovery and optimisation.
- Functions:
– Management of hypothesis databases.
– Versioning of hypotheses
– Tracking of hypothesis evaluation history - Supported technologies:
– Jupyter Notebooks (management of experimental results)
– Git, DVC (data versioning)
– SQLite, NoSQL (hypothesis database)
10. user interface (UI) module
- Role: to present the results of the hypothesis generation system in an easy-to-understand way for the user, to enable interactive operation, to visualise the results of hypothesis proposal and testing, and to provide a mechanism for the user to give feedback.
- Functions.
– Visualisation of hypothesis generation results
– Collection of user feedback
– Manipulation and re-generation of hypotheses - Supported technologies:
– React, Angular (front-end framework)
– Flask, Django (back-end development)
– Dash, Streamlit (data science visualisation tools)
The overall workflow of the system is as follows: data collection → data pre-processing → exploratory data analysis (EDA) → machine learning/natural language processing → causal inference/generative modelling → hypothesis evaluation → hypothesis management and tracking → results display in the user interface.
This establishes a flow where hypotheses are automatically generated from the data and the validity of the hypotheses is managed and evaluated throughout the system.
Specific application examples
Specific examples of applications are described below.
1. medical field: new drug development
Description: a system that analyses large-scale medical data, genetic information and past research papers to automatically generate hypotheses for the discovery and development of new drugs. AI automatically proposes hypotheses in a data-driven manner to search for drug candidates and their relevance to disease, which would take an enormous amount of time using conventional methods.
Application example.
– Disease-gene association hypothesis: AI generates a hypothesis on whether a particular gene is the cause of a certain disease based on genetic information, and then proposes a new drug target based on that hypothesis.
– Drug repurposing hypotheses: a hypothesis generation system is used to automatically test whether an existing drug is effective for another disease, and to discover new indications.
Real-world examples: companies such as Insilico Medicine* are using AI to build hypothesis-generating systems to discover new treatments, thereby dramatically reducing the time to discovery of drug candidates.
2. financial sector: stock market forecasting
Description: Analyses stock market and financial instrument trading data to generate hypotheses on future market trends and risks; AI analyses large-scale market data, news data and social media information to provide hypotheses to support investment strategies.
Application examples
– Automatic generation of stock price fluctuation hypotheses: AI generates hypotheses on future stock price trends and risks based on historical stock market data, economic indicators and news. Investors can make investment decisions based on these.
– Causal hypothesis of market events: hypothesises the impact of a specific event (e.g. corporate earnings or political decision) on stock prices and the market as a whole and estimates the magnitude of that impact.
Real-world example: Kensho Technologies provides a system that analyses large amounts of market data and news and automatically generates investment-relevant hypotheses, thereby helping traders and analysts to make decisions more quickly.
3. energy sector: smart grid optimisation
Description: In optimising energy consumption and introducing renewable energy, AI analyses power generation, consumption patterns and weather data to automatically generate hypotheses and improve energy management systems.
Application examples
– Hypothesis generation for energy consumption patterns: based on consumer behaviour and weather data, AI can predict energy demand and propose hypotheses for energy-saving measures to realise efficient energy supply.
– Hypothesis generation forecasting for renewable energy: predicts the amount of electricity generated by solar and wind power using weather data, and hypothesises optimal electricity supply strategies based on this.
Real-world example: the National Renewable Energy Laboratory (NREL) uses AI-based hypothesis generation to forecast renewable energy generation and optimise the energy supply balance.
4. automotive sector: improving safety in automated driving
Description: To improve the safety of automated vehicles, systems in which AI generates hypotheses on abnormality detection and hazard prediction based on driving data and sensor information. The automated driving system enhances safety by responding to abnormal situations based on the hypotheses.
Application examples
– Accident risk prediction hypotheses: by analysing vehicle and environmental data, AI generates hypotheses for the risk of accidents that may occur under certain conditions. This allows adjustments to be made so that preventive systems can operate.
– Driving pattern optimisation hypothesis: a hypothesis is generated for the vehicle to take an optimum route or driving pattern to improve fuel efficiency and driving efficiency.
Real-world examples: self-driving developers such as Waymo and Tesla use AI to generate hypotheses on accident risks and abnormal behaviour from vast amounts of driving data to improve safety.
5. manufacturing industry: anomaly detection in production lines
Description: In the manufacturing industry, AI is being introduced to detect and improve abnormalities and quality problems on production lines. Based on real-time data from sensors and past production data, AI generates hypotheses on the causes of abnormalities and preventive measures.
Application examples
– Prediction hypothesis for equipment breakdowns: data obtained from sensors is analysed to generate hypotheses for predicting machine breakdowns, and based on these hypotheses, maintenance is carried out in advance to reduce downtime.
– Hypotheses for improving product quality: based on production process data, AI proposes hypotheses on optimal conditions and processes to improve product quality.
Real-world example: General Electric (GE) has introduced an AI-based abnormality detection system that automatically generates hypotheses for the early detection of faults and defects on production lines to improve maintenance efficiency.
Challenges and countermeasures
The challenges and measures to be taken to put the automatic hypothesis generation system into practical use are described below.
1. data quality and quantity
Challenge: automatic hypothesis generation requires large amounts of high-quality data, but if the data that can be collected is insufficient or of low quality, the accuracy of the hypotheses generated may be reduced.
Solution:
– Diversify data collection: collect data from a variety of sources through web scraping, APIs and data collection from databases.
– Data cleaning: build automated pre-processing pipelines to improve data quality by removing noise and completing missing values.
2. over-training of models
Challenge: complex models over-adapt to training data, reducing their ability to generalise to unknown data.
Solution:
– Apply regularisation methods: use methods such as L1/L2 regularisation and drop-out to reduce model complexity.
– Cross-validation: divide the data into training and validation sets to evaluate the model and detect over-learning.
3. lack of interpretability
Challenge: users may doubt the reliability of automatically generated hypotheses if the basis for them is unknown.
Solution:
– Implement visualisation tools: visualisation of the data behind the model’s decision criteria and hypotheses to aid understanding.
– Use of Explainable AI (XAI): use LIME as described in ‘Explainable Artificial Intelligence (13) Model-independent Interpretations (Local Surrogate :LIME)’ and SHAP (SHapley Additive exPlanations) as described in ‘Explainable Artificial Intelligence (16) Model-independent Interpretations (SHAP) SHAP (SHapley Additive exPlanations))’ to make the model predictions interpretable.
4. implementation costs and resources
Challenge: the development of automatic hypothesis generation systems requires specialist knowledge and skills and may be under-resourced.
Solution:
– Use of open source libraries: reduce development costs by utilising existing libraries and tools (e.g. TensorFlow, PyTorch, Scikit-learn).
– Use of cloud computing: use cloud services such as AWS and Google Cloud to flexibly manage resources and ensure the required computing power.
5. ethical and legal issues
Challenge: ethical concerns can arise about the impact of automatically generated hypotheses on humans and society.
Solution:
– Develop ethical guidelines: develop ethical guidelines for the design and operation of automated hypothesis generation systems and make them transparent.
– Assess social impact: assess the social impact of the hypotheses during the development phase and make modifications as necessary.
6. user interaction
Challenge: automatically generated hypotheses are less likely to be accepted if they do not match user needs and expectations.
Solution:
– Collect feedback from users: design an interface that allows users to evaluate hypotheses and provide feedback, which can be used for improvement.
– User education: educate users on how to use the system and the background to the hypotheses generated, to increase user understanding and acceptance.
References
References relevant to automatic hypothesis generation systems are listed below.
1. literature on automated hypothesis generation
– Title: ‘Automated Hypothesis Generation in Scientific Research’.
Authors: V. D. S. K. Reddy, A. D. Tharun, et al.
Source: Journal of Computational Science, 2020.
Description: describes frameworks and algorithms for automatic hypothesis generation.
– Title: ‘Hypothesis Generation through Data Mining in Biomedical Research’.
2. machine learning and data analysis
– Title: ‘Pattern Recognition and Machine Learning’.
Authors: Christopher M. Bishop
Publisher: Springer, 2006.
Description: a comprehensive text on the theory and applications of machine learning.
– Title: ‘Deep Learning’.
Authors: Ian Goodfellow, Yoshua Bengio, and Aaron Courville
Publisher: MIT Press, 2016.
Description: a detailed account of the theory and practice of deep learning.
3. natural language processing (NLP)
– Title: ‘Speech and Language Processing’.
Authors: Daniel Jurafsky and James H. Martin
Publisher: Prentice Hall, 2008.
Description: An important resource on the fundamentals and applications of natural language processing.
4. ethical considerations
– Title: “”
5. related software and frameworks
– Title: ‘Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow’
Authors: Aurélien Géron.
Publisher: O’Reilly Media, 2019.
Description: hands-on guide to learning how to implement machine learning in practice.
6. case study
– Title: “Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomy“
コメント