problem solving agent algorithm

• Trending Now
• Foundational Courses
• Data Science
• Practice Problem
• Machine Learning
• System Design
• DevOps Tutorial
• Statistical Machine Translation of Languages in Artificial Intelligence
• Breadth-first Search is a special case of Uniform-cost search
• Artificial Intelligence - Boon or Bane
• Stochastic Games in Artificial Intelligence
• Resolution Algorithm in Artificial Intelligence
• Types of Environments in AI
• PEAS Description of Task Environment
• Optimal Decision Making in Multiplayer Games
• Game Theory in AI
• Emergence Of Artificial Intelligence
• Propositional Logic based Agent
• GPT-3 : Next AI Revolution
• Advantages and Disadvantage of Artificial Intelligence
• Understanding PEAS in Artificial Intelligence
• Sparse Rewards in Reinforcement Learning
• Propositional Logic Hybrid Agent and Logical State
• Prepositional Logic Inferences
• Linguistic variable And Linguistic hedges
• Knowledge based agents in AI

Problem Solving in Artificial Intelligence

The reflex agent of AI directly maps states into action. Whenever these agents fail to operate in an environment where the state of mapping is too large and not easily performed by the agent, then the stated problem dissolves and sent to a problem-solving domain which breaks the large stored problem into the smaller storage area and resolves one by one. The final integrated action will be the desired outcomes.

On the basis of the problem and their working domain, different types of problem-solving agent defined and use at an atomic level without any internal state visible with a problem-solving algorithm. The problem-solving agent performs precisely by defining problems and several solutions. So we can say that problem solving is a part of artificial intelligence that encompasses a number of techniques such as a tree, B-tree, heuristic algorithms to solve a problem.  

We can also say that a problem-solving agent is a result-driven agent and always focuses on satisfying the goals.

There are basically three types of problem in artificial intelligence:

1. Ignorable: In which solution steps can be ignored.

2. Recoverable: In which solution steps can be undone.

3. Irrecoverable: Solution steps cannot be undo.

Steps problem-solving in AI: The problem of AI is directly associated with the nature of humans and their activities. So we need a number of finite steps to solve a problem which makes human easy works.

These are the following steps which require to solve a problem :

• Problem definition: Detailed specification of inputs and acceptable system solutions.
• Problem analysis: Analyse the problem thoroughly.
• Knowledge Representation: collect detailed information about the problem and define all possible techniques.
• Problem-solving: Selection of best techniques.

Components to formulate the associated problem: 

• Initial State: This state requires an initial state for the problem which starts the AI agent towards a specified goal. In this state new methods also initialize problem domain solving by a specific class.
• Action: This stage of problem formulation works with function with a specific class taken from the initial state and all possible actions done in this stage.
• Transition: This stage of problem formulation integrates the actual action done by the previous action stage and collects the final stage to forward it to their next stage.
• Goal test: This stage determines that the specified goal achieved by the integrated transition model or not, whenever the goal achieves stop the action and forward into the next stage to determines the cost to achieve the goal.  
• Path costing: This component of problem-solving numerical assigned what will be the cost to achieve the goal. It requires all hardware software and human working cost.

Please Login to comment...

Similar reads.

• Artificial Intelligence
• 5 Reasons to Start Using Claude 3 Instead of ChatGPT
• 6 Ways to Identify Who an Unknown Caller
• 10 Best Lavender AI Alternatives and Competitors 2024
• The 7 Best AI Tools for Programmers to Streamline Development in 2024
• 30 OOPs Interview Questions and Answers (2024)

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

Problem Solving Agents in Artificial Intelligence

In this post, we will talk about Problem Solving agents in Artificial Intelligence, which are sort of goal-based agents. Because the straight mapping from states to actions of a basic reflex agent is too vast to retain for a complex environment, we utilize goal-based agents that may consider future actions and the desirability of outcomes.

You Will Learn

Problem Solving Agents

Problem Solving Agents decide what to do by finding a sequence of actions that leads to a desirable state or solution.

An agent may need to plan when the best course of action is not immediately visible. They may need to think through a series of moves that will lead them to their goal state. Such an agent is known as a problem solving agent , and the computation it does is known as a search .

The problem solving agent follows this four phase problem solving process:

• Goal Formulation: This is the first and most basic phase in problem solving. It arranges specific steps to establish a target/goal that demands some activity to reach it. AI agents are now used to formulate goals.
• Problem Formulation: It is one of the fundamental steps in problem-solving that determines what action should be taken to reach the goal.
• Search: After the Goal and Problem Formulation, the agent simulates sequences of actions and has to look for a sequence of actions that reaches the goal. This process is called search, and the sequence is called a solution . The agent might have to simulate multiple sequences that do not reach the goal, but eventually, it will find a solution, or it will find that no solution is possible. A search algorithm takes a problem as input and outputs a sequence of actions.
• Execution: After the search phase, the agent can now execute the actions that are recommended by the search algorithm, one at a time. This final stage is known as the execution phase.

Problems and Solution

Before we move into the problem formulation phase, we must first define a problem in terms of problem solving agents.

A formal definition of a problem consists of five components:

Initial State

Transition model.

It is the agent’s starting state or initial step towards its goal. For example, if a taxi agent needs to travel to a location(B), but the taxi is already at location(A), the problem’s initial state would be the location (A).

It is a description of the possible actions that the agent can take. Given a state s, Actions ( s ) returns the actions that can be executed in s. Each of these actions is said to be appropriate in s.

It describes what each action does. It is specified by a function Result ( s, a ) that returns the state that results from doing action an in state s.

The initial state, actions, and transition model together define the state space of a problem, a set of all states reachable from the initial state by any sequence of actions. The state space forms a graph in which the nodes are states, and the links between the nodes are actions.

It determines if the given state is a goal state. Sometimes there is an explicit list of potential goal states, and the test merely verifies whether the provided state is one of them. The goal is sometimes expressed via an abstract attribute rather than an explicitly enumerated set of conditions.

It assigns a numerical cost to each path that leads to the goal. The problem solving agents choose a cost function that matches its performance measure. Remember that the optimal solution has the lowest path cost of all the solutions .

Example Problems

The problem solving approach has been used in a wide range of work contexts. There are two kinds of problem approaches

• Standardized/ Toy Problem: Its purpose is to demonstrate or practice various problem solving techniques. It can be described concisely and precisely, making it appropriate as a benchmark for academics to compare the performance of algorithms.
• Real-world Problems: It is real-world problems that need solutions. It does not rely on descriptions, unlike a toy problem, yet we can have a basic description of the issue.

Some Standardized/Toy Problems

Vacuum world problem.

Let us take a vacuum cleaner agent and it can move left or right and its jump is to suck up the dirt from the floor.

The vacuum world’s problem can be stated as follows:

States: A world state specifies which objects are housed in which cells. The objects in the vacuum world are the agent and any dirt. The agent can be in either of the two cells in the simple two-cell version, and each call can include dirt or not, therefore there are 2×2×2 = 8 states. A vacuum environment with n cells has n×2 n states in general.

Initial State: Any state can be specified as the starting point.

Actions: We defined three actions in the two-cell world: sucking, moving left, and moving right. More movement activities are required in a two-dimensional multi-cell world.

Transition Model: Suck cleans the agent’s cell of any filth; Forward moves the agent one cell forward in the direction it is facing unless it meets a wall, in which case the action has no effect. Backward moves the agent in the opposite direction, whilst TurnRight and TurnLeft rotate it by 90°.

Goal States: The states in which every cell is clean.

Action Cost: Each action costs 1.

8 Puzzle Problem

In a sliding-tile puzzle , a number of tiles (sometimes called blocks or pieces) are arranged in a grid with one or more blank spaces so that some of the tiles can slide into the blank space. One variant is the Rush Hour puzzle, in which cars and trucks slide around a 6 x 6 grid in an attempt to free a car from the traffic jam. Perhaps the best-known variant is the 8- puzzle (see Figure below ), which consists of a 3 x 3 grid with eight numbered tiles and one blank space, and the 15-puzzle on a 4 x 4  grid. The object is to reach a specified goal state, such as the one shown on the right of the figure. The standard formulation of the 8 puzzles is as follows:

STATES : A state description specifies the location of each of the tiles.

INITIAL STATE : Any state can be designated as the initial state. (Note that a parity property partitions the state space—any given goal can be reached from exactly half of the possible initial states.)

ACTIONS : While in the physical world it is a tile that slides, the simplest way of describing action is to think of the blank space moving Left , Right , Up , or Down . If the blank is at an edge or corner then not all actions will be applicable.

TRANSITION MODEL : Maps a state and action to a resulting state; for example, if we apply Left to the start state in the Figure below, the resulting state has the 5 and the blank switched.

GOAL STATE :  It identifies whether we have reached the correct goal state. Although any state could be the goal, we typically specify a state with the numbers in order, as in the Figure above.

ACTION COST : Each action costs 1.

You Might Like:

• Agents in Artificial Intelligence

Types of Environments in Artificial Intelligence

• Understanding PEAS in Artificial Intelligence
• River Crossing Puzzle | Farmer, Wolf, Goat and Cabbage

Share Article:

Digital image processing: all you need to know.

Problem-Solving Agents In Artificial Intelligence

In artificial intelligence, a problem-solving agent refers to a type of intelligent agent designed to address and solve complex problems or tasks in its environment. These agents are a fundamental concept in AI and are used in various applications, from game-playing algorithms to robotics and decision-making systems. Here are some key characteristics and components of a problem-solving agent:

• Perception : Problem-solving agents typically have the ability to perceive or sense their environment. They can gather information about the current state of the world, often through sensors, cameras, or other data sources.
• Knowledge Base : These agents often possess some form of knowledge or representation of the problem domain. This knowledge can be encoded in various ways, such as rules, facts, or models, depending on the specific problem.
• Reasoning : Problem-solving agents employ reasoning mechanisms to make decisions and select actions based on their perception and knowledge. This involves processing information, making inferences, and selecting the best course of action.
• Planning : For many complex problems, problem-solving agents engage in planning. They consider different sequences of actions to achieve their goals and decide on the most suitable action plan.
• Actuation : After determining the best course of action, problem-solving agents take actions to interact with their environment. This can involve physical actions in the case of robotics or making decisions in more abstract problem-solving domains.
• Feedback : Problem-solving agents often receive feedback from their environment, which they use to adjust their actions and refine their problem-solving strategies. This feedback loop helps them adapt to changing conditions and improve their performance.
• Learning : Some problem-solving agents incorporate machine learning techniques to improve their performance over time. They can learn from experience, adapt their strategies, and become more efficient at solving similar problems in the future.

Problem-solving agents can vary greatly in complexity, from simple algorithms that solve straightforward puzzles to highly sophisticated AI systems that tackle complex, real-world problems. The design and implementation of problem-solving agents depend on the specific problem domain and the goals of the AI application.

Hello, I’m Hridhya Manoj. I’m passionate about technology and its ever-evolving landscape. With a deep love for writing and a curious mind, I enjoy translating complex concepts into understandable, engaging content. Let’s explore the world of tech together

Which Of The Following Is A Privilege In SQL Standard

Implicit Return Type Int In C

Leave a Comment Cancel reply

Save my name, email, and website in this browser for the next time I comment.

Reach Out to Us for Any Query

SkillVertex is an edtech organization that aims to provide upskilling and training to students as well as working professionals by delivering a diverse range of programs in accordance with their needs and future aspirations.

© 2024 Skill Vertex

• Speakers & Mentors
• AI services

Learn About Problem Solving Agents in Artificial Intelligence on Tutorialspoint

Artificial intelligence is a rapidly growing field that aims to develop computer systems capable of performing tasks that typically require human intelligence. One of the key areas in AI is problem solving, where agents are designed to find solutions to complex problems.

TutorialsPoint provides an in-depth tutorial on problem solving agents in artificial intelligence, covering various techniques and algorithms used to tackle different types of problems. Whether you are a beginner or an experienced AI practitioner, this tutorial will equip you with the knowledge and skills needed to build effective problem solving agents.

In this tutorial, you will learn about different problem solving frameworks, such as the goal-based approach, the utility-based approach, and the constraint satisfaction approach. You will also explore various search algorithms, including uninformed search algorithms like depth-first search and breadth-first search, as well as informed search algorithms like A* search and greedy search.

Problem solving agents in artificial intelligence play a crucial role in many real-world applications, ranging from robotics and automation to data analysis and decision-making systems. By mastering the concepts and techniques covered in this tutorial, you will be able to design and develop intelligent agents that can effectively solve complex problems.

What are Problem Solving Agents?

In the field of artificial intelligence, problem solving agents are intelligent systems that are designed to solve complex problems. These agents are equipped with the ability to analyze a given problem, search for possible solutions, and select the best solution based on a set of defined criteria.

Problem solving agents can be thought of as entities that can interact with their environment and take actions to achieve a desired goal. These agents are typically equipped with sensors to perceive the environment, an internal representation of the problem, and actuators to take actions.

One of the key challenges in designing problem solving agents is to define a suitable representation of the problem and its associated constraints. This representation allows the agent to reason about the problem and generate potential solutions. The agent can then evaluate these solutions and select the one that is most likely to achieve the desired goal.

Problem solving agents can be encountered in various domains, such as robotics, computer vision, natural language processing, and even in game playing. These agents can be implemented using different techniques, including search algorithms, constraint satisfaction algorithms, and machine learning algorithms.

In conclusion, problem solving agents are intelligent systems that are designed to solve complex problems by analyzing the environment, searching for solutions, and selecting the best solution based on a set of defined criteria. These agents can be encountered in various domains and can be implemented using different techniques.

Types of Problem Solving Agents

In the field of artificial intelligence, problem solving agents are designed to tackle a wide range of issues and tasks. These agents are built to analyze problems, explore potential solutions, and make decisions based on their findings. Here, we will explore some of the common types of problem solving agents.

Simple Reflex Agents

A simple reflex agent is a basic type of problem solving agent that relies on a set of predefined rules or conditions to make decisions. These rules are typically in the form of “if-then” statements, where the agent takes certain actions based on the current state of the problem. Simple reflex agents are often used in situations where the problem can be easily mapped to a small set of conditions.

Model-Based Reflex Agents

A model-based reflex agent goes beyond simple reflex agents by maintaining an internal model of the problem and its environment. This model allows the agent to have a better understanding of its current state and make more informed decisions. Model-based reflex agents use the current state and the model to determine the appropriate action to take. These agents are often used in more complex problem-solving scenarios.

Goal-Based Agents

A goal-based agent is designed to achieve a specific goal or set of goals. These agents analyze the current state of the problem and then determine a sequence of actions that will lead to the desired outcome. Goal-based agents often use search algorithms to explore the possible paths and make decisions based on their analysis. These agents are commonly used in planning and optimization problems.

Utility-Based Agents

Utility-based agents make decisions based on a utility function or a measure of the desirability of different outcomes. These agents assign a value or utility to each possible action and choose the action that maximizes the overall utility. Utility-based agents are commonly used in decision-making problems where there are multiple possible outcomes with varying levels of desirability.

These are just a few examples of the types of problem solving agents that can be found in the field of artificial intelligence. Each type of agent has its own strengths and weaknesses and is suited to different problem-solving scenarios. By understanding the different types of agents, developers and researchers can choose the most appropriate agent for their specific problem and improve the efficiency and effectiveness of their problem-solving solutions.

Components of Problem Solving Agents

A problem-solving agent is a key concept in the field of artificial intelligence (AI). It is an agent that can analyze a given problem and take appropriate actions to solve it. The agents are designed using a set of components that work together to achieve their goals.

One of the main components of a problem-solving agent is the solving component. This component is responsible for applying different algorithms and techniques to find the best solution to a given problem. The solving component can use various approaches such as search algorithms, constraint satisfaction, optimization techniques, and machine learning.

TutorialsPoint is a popular online platform that offers a wealth of resources on various topics, including artificial intelligence and problem-solving agents. These tutorials provide step-by-step instructions and examples to help learners understand and implement different problem-solving techniques.

Another important component of a problem-solving agent is the knowledge component. This component stores the agent’s knowledge about the problem domain, including facts, rules, and constraints. The knowledge component is crucial for guiding the agent’s problem-solving process and making informed decisions.

The problem component is responsible for representing and defining the problem that the agent needs to solve. It includes information such as the initial state, goal state, and possible actions that the agent can take. The problem component provides the necessary context for the agent to analyze and solve the problem effectively.

Finally, the agents component is responsible for coordinating the activities of different components and controlling the overall behavior of the problem-solving agent. It receives inputs from the environment, communicates with the other components, and takes actions based on the current state of the problem. The agents component plays a crucial role in ensuring the problem-solving agent operates efficiently and effectively.

In conclusion, problem-solving agents in artificial intelligence are designed using various components such as solving, knowledge, problem, and agents. These components work together to analyze a problem, apply appropriate techniques, and find the best solution. TutorialsPoint is a valuable resource for learning about different problem-solving techniques and implementing them in practice.

Search Strategies for Problem Solving Agents

Intelligence is a complex and fascinating field that encompasses a wide range of topics and technologies. One area of focus in artificial intelligence is problem solving. Problem solving agents are designed to find solutions to specific problems by searching through a large space of possible solutions.

TutorialsPoint provides valuable resources and tutorials on problem solving agents in artificial intelligence. These tutorials cover various search strategies that can be employed by problem solving agents to efficiently find optimal solutions.

Search strategies play a crucial role in the efficiency and effectiveness of problem solving agents. Some common search strategies include:

These are just a few examples of search strategies that problem solving agents can utilize. The choice of search strategy depends on the specific problem at hand and the available resources.

TutorialsPoint’s comprehensive tutorials on problem solving agents in artificial intelligence provide in-depth explanations, examples, and implementation guides for each search strategy. By leveraging these tutorials, developers and researchers can enhance their understanding of problem solving agents and apply them to real-world scenarios.

Uninformed Search Algorithms

Uninformed search algorithms are a category of algorithms used by problem-solving agents in artificial intelligence. These algorithms do not have any information about the problem domain and make decisions solely based on the current state and possible actions.

Breadth-First Search

Breadth-first search is one of the basic uninformed search algorithms. It explores all the neighbor nodes at the present depth before moving on to nodes at the next depth level. It guarantees the shortest path to the goal state if there is one, but it can be inefficient for large search spaces.

Depth-First Search

Depth-first search is another uninformed search algorithm. It explores a path all the way to the deepest level before backtracking. It is often implemented using a stack data structure. Depth-first search is not guaranteed to find the shortest path to the goal state, but it can be more memory-efficient than breadth-first search.

Uninformed search algorithms are widely used in problem-solving scenarios where there is no additional information available about the problem domain. They are important tools in the field of artificial intelligence and are taught in many tutorials and courses, including the ones available on TutorialsPoint.

Breadth First Search (BFS)

Breadth First Search (BFS) is a fundamental algorithm used in artificial intelligence for problem solving. It is a graph traversal algorithm that explores all vertices of a graph in a breadthward motion, starting from a given source vertex.

BFS is commonly used to solve problems in AI, such as finding the shortest path between two nodes, determining if a graph is connected, or generating all possible solutions to a problem.

In BFS, the algorithm continuously explores the vertices adjacent to the current vertex before moving on to the next level of vertices. This approach ensures that all vertices at each level are visited before proceeding to the next level. The algorithm uses a queue to keep track of the vertices that need to be visited.

The main steps of the BFS algorithm are as follows:

• Choose a source vertex and mark it as visited.
• Enqueue the source vertex.
• While the queue is not empty, dequeue a vertex and visit it.
• Enqueue all the adjacent vertices of the visited vertex that have not been visited before.
• Mark the dequeued vertex as visited.

By following these steps, BFS explores the graph level by level, guaranteeing that the shortest path from the source vertex to any other vertex is found.

BFS has a time complexity of O(V + E), where V is the number of vertices and E is the number of edges in the graph. It is considered an efficient algorithm for solving problems in artificial intelligence.

Depth First Search (DFS)

Depth First Search (DFS) is a popular graph traversal algorithm commonly used in artificial intelligence and problem solving agents. It is also commonly used in tutorialspoint for teaching the concepts of graph traversal algorithms.

DFS starts at a given node of a graph and explores as far as possible along each branch before backtracking. It uses a stack data structure to keep track of the nodes to be visited. The algorithm visit nodes in a depthward motion, meaning that it explores the deepest paths in the graph first.

The DFS algorithm can be implemented as follows:

• Start at the given node and mark it as visited.
• If the node has unvisited neighbors, choose one and push it onto the stack.
• If there are no unvisited neighbors, backtrack by popping the stack.
• If the stack is empty, the algorithm terminates.
• Repeat steps 2-4 until all nodes are visited.

DFS is often used to solve problems that can be represented as graphs, such as finding solutions in a maze or searching for a path between two points. It can also be used to perform topological sorting and cycle detection in directed graphs.

DFS has a few advantages over other graph traversal algorithms. It requires less memory than breadth-first search (BFS) because it only needs to store the visited nodes in a stack. It can also be easily implemented recursively.

However, DFS may not find the shortest path between two nodes in a graph, as it explores deep paths first. It can also get stuck in infinite loops if the graph contains cycles.

Overall, DFS is a powerful algorithm that can be used to solve a wide range of problems in artificial intelligence and problem solving agents. By understanding how it works, developers can effectively apply it to various scenarios in their projects.

Iterative Deepening Depth First Search (IDDFS)

The Iterative Deepening Depth First Search (IDDFS) is a technique used in artificial intelligence to solve problems efficiently. It is a combination of depth-first search (DFS) and breadth-first search (BFS) algorithms.

The IDDFS algorithm starts with a depth limit of 0 and gradually increases the depth limit until the goal is found or all possibilities have been explored. It works by performing a depth-first search up to the current depth limit, and if the goal is not found, it resets the visited nodes and increases the depth limit by 1.

This approach allows the IDDFS algorithm to explore the search space more efficiently compared to standard depth-first search. It avoids the disadvantages of BFS, such as high memory usage, by only keeping track of the current path.

The IDDFS algorithm is particularly useful in situations where the search space is large and the goal is likely to be found at a relatively small depth. It guarantees that the algorithm will find the solution if it exists within the search space.

Overall, the IDDFS algorithm is a powerful tool in problem-solving agents in artificial intelligence. It combines the advantages of depth-first search and breadth-first search, making it an efficient and effective approach for solving complex problems.

Uniform Cost Search (UCS)

Uniform Cost Search (UCS) is a problem-solving algorithm used in the field of artificial intelligence. It is a variant of the general graph search algorithm, which aims to find the cheapest path from a starting node to a goal node in a weighted graph.

In UCS, each action in the problem domain has a cost associated with it. The algorithm expands the nodes in a graph in a cost-effective manner, always choosing the node with the lowest cost so far. This ensures that the optimal solution with the minimum total cost is found.

The UCS algorithm maintains a priority queue of nodes, where the priority is based on the accumulated cost to reach each node. Initially, the start node is inserted into the priority queue with a cost of zero. The algorithm then iteratively selects and expands the node with the lowest cost, updating the priority queue accordingly.

During the expansion process, the algorithm checks if the goal node has been reached. If not, it generates the successor nodes of the expanded node and adds them to the priority queue. The cost of each successor node is calculated by adding the cost of the action that led to that node to the total cost so far. This process continues until the goal node is reached.

Uniform Cost Search is considered to be complete and optimal, meaning it will always find a solution if one exists, and it will find the optimal solution with the minimum cost. However, it can be computationally expensive, especially in large graphs or graphs with high branching factors.

In summary, Uniform Cost Search is a powerful problem-solving algorithm used in artificial intelligence to find the cheapest path from a starting node to a goal node in a weighted graph. By prioritizing nodes based on their accumulated cost, UCS ensures that the optimal solution with the minimum total cost is found.

Informed Search Algorithms

In the field of Artificial Intelligence, informed search algorithms are a group of problem-solving agents that use knowledge or information about the problem domain to guide the search process. Unlike uninformed search algorithms, which do not have any additional information about the problem, informed search algorithms make use of heuristics or other measures of desirability to guide the search towards the goal state more efficiently.

One of the most well-known informed search algorithms is the A* algorithm. The A* algorithm uses a combination of the cost to reach a certain state and an estimate of the cost from that state to the goal state, known as the heuristic function. By selecting the states with the lowest total cost, the A* algorithm can efficiently navigate through the search space and find the optimal solution.

Another popular informed search algorithm is the greedy best-first search. Greedy best-first search evaluates each state based solely on the heuristic estimate of the cost to reach the goal state. It always chooses the state that appears to be the closest to the goal, without considering the overall cost. Although this algorithm is efficient for some problems, it can also get stuck in local optima and fail to find the optimal solution.

In addition to A* and greedy best-first search, there are various other informed search algorithms, such as the iterative deepening A* (IDA*) algorithm and the simulated annealing algorithm. Each algorithm has its strengths and weaknesses and is suited for different types of problems.

In conclusion, informed search algorithms play an important role in problem-solving in the field of Artificial Intelligence. They use additional knowledge or information about the problem domain to guide the search process and find optimal solutions more efficiently. By considering both the cost to reach a certain state and an estimate of the cost to the goal state, these algorithms can effectively navigate through the search space and find the best possible solution.

Heuristic Functions

In the field of artificial intelligence, heuristic functions play a crucial role in problem solving. A heuristic function is a function that estimates the cost or utility of reaching a goal state from a given state in a problem. It provides a way for agents to make informed decisions about their actions while navigating through a problem-solving process.

Heuristic functions are designed to guide the search process towards the most promising directions in order to find a solution efficiently. They provide a measure of how close a particular state is to the goal state, without having complete information about the problem domain. This allows the agent to prioritize its actions and choose the most promising ones at each step.

When designing a heuristic function, it is important to consider the specific problem at hand and the available knowledge about it. The function should be able to assess the cost or utility of reaching the goal state based on the available information and the characteristics of the problem. It should also be computationally efficient to ensure that it can be used in real-time problem-solving scenarios.

The effectiveness of a heuristic function depends on its ability to accurately estimate the cost or utility of reaching the goal state. A good heuristic function should provide a tight lower bound on the actual cost, meaning that it should never overestimate the distance to the goal. This allows the agent to efficiently explore the solution space and reach the goal state optimally.

In conclusion, heuristic functions are essential tools in artificial intelligence for problem solving. They enable agents to make informed decisions and guide the search process towards the most promising directions. By accurately estimating the cost or utility of reaching the goal state, heuristic functions help agents find optimal solutions efficiently.

A* Algorithm

The A* algorithm is a commonly used artificial intelligence technique for solving problems. It is particularly useful in problem-solving agents that aim to find the optimal path or solution. With the A* algorithm, an agent can navigate through a problem space, evaluating and selecting the most promising paths based on heuristic estimates and actual costs.

A* combines the advantages of both uniform cost search and best-first search algorithms. It uses a heuristic function to estimate the cost from the current state to the goal state, allowing the agent to prioritize its search and explore the most promising paths first. The heuristic function provides an estimation of the remaining cost, often referred to as the “H-cost”.

The algorithm maintains a priority queue, also known as an open list, that keeps track of the states or nodes to be expanded. At each step, the agent selects the state with the lowest combined cost, known as the “F-cost”, which is calculated as the sum of the actual cost from the starting state to the current state (known as the “G-cost”) and the heuristic cost.

The A* algorithm guarantees to find the optimal path if certain conditions are met. The heuristic function must be admissible, meaning that it never overestimates the actual cost. Additionally, the function should be consistent or monotonic, which ensures that the heuristic estimate is always less than or equal to the cost of reaching the goal state from the current state.

In conclusion, the A* algorithm is a powerful tool used in artificial intelligence for solving problems. It combines the benefits of both uniform cost search and best-first search algorithms, making it an efficient and effective choice for problem-solving agents. By prioritizing the most promising paths, as estimated by the heuristic function, the A* algorithm can find the optimal solution in a timely manner.

Greedy Best-first Search

The Greedy Best-first Search is a type of problem solving algorithm that is used in artificial intelligence. It is an informed search algorithm that uses heuristics to guide its search through a problem space.

The goal of the Greedy Best-first Search is to find a solution to a problem by always choosing the most promising path at each step. It does this by evaluating the estimated cost of each possible next step based on a heuristic function. The heuristic function provides an estimate of how close the current state is to the goal state.

This algorithm is called “greedy” because it always chooses the path that appears to be the best at the current moment, without considering the future consequences. It does not take into account the possibility that a different path might lead to a better solution in the long run.

Despite its simplicity, the Greedy Best-first Search can be quite effective in certain types of problems. However, it has some limitations. Since it only considers the local information at each step, it may overlook better paths that require more steps to reach the goal. In addition, it may get stuck in loops or infinite cycles if there are no better paths to follow.

Algorithm Steps:

• Initialize the search with the initial state.
• Choose the most promising state based on the heuristic function.
• Move to the chosen state.
• If the chosen state is the goal state, stop the search.

In conclusion, the Greedy Best-first Search is a simple yet effective problem solving algorithm in artificial intelligence. It can be a useful tool when solving certain types of problems, but it also has its limitations. It is important to understand the strengths and weaknesses of different search algorithms when applying them to real-world problems.

Hill Climbing

The Hill Climbing algorithm is a popular search algorithm used in artificial intelligence to solve problem-solving tasks. It is commonly used in optimization problems where the goal is to find the best possible solution among a set of alternatives.

In hill climbing, the agent starts with an initial solution and iteratively makes small improvements to the solution by locally exploring the neighboring states. The agent selects the next state to move to based on an evaluation function, which assigns a value to each state indicating its desirability. The agent moves to the state with the highest value and repeats the process until no better state can be found.

The hill climbing algorithm can be thought of as climbing a hill, where the agent starts at the bottom and tries to reach the highest point by taking steps uphill. However, hill climbing can get stuck at local optima, which are points that are higher than their immediate neighbors but lower than the global optimum. This means that the algorithm may fail to find the best solution if it gets trapped in a local maximum.

Types of Hill Climbing

There are different variations of the hill climbing algorithm that address its limitations:

Steepest Ascent Hill Climbing

The steepest ascent hill climbing algorithm is a variation of hill climbing where the agent considers all possible moves from the current state and selects the one that leads to the state with the highest value. This approach ensures that the agent moves uphill as much as possible in each iteration, but it can be computationally expensive as it needs to evaluate all possible moves.

First-Choice Hill Climbing

The first-choice hill climbing algorithm is another variation of hill climbing where the agent randomly selects one move from the current state and moves to the state with the highest value among the randomly selected moves. This approach can be more efficient than steepest ascent hill climbing as it does not need to evaluate all possible moves, but it may still get stuck in local optima.

In conclusion, hill climbing is a widely used algorithm in artificial intelligence for solving problem-solving tasks. Despite its limitations, it can be effective in finding good solutions for optimization problems. However, it is important to be aware of the possibility of getting stuck in local optima and consider using other algorithms or techniques to overcome this limitation.

Simulated Annealing

Simulated Annealing is a problem-solving algorithm used in artificial intelligence. It is a probabilistic method that is inspired by the annealing process used in metallurgy.

The goal of simulated annealing is to find a good solution to a problem, even when the search space is large and the problem is difficult to solve. It starts with an initial solution and gradually explores the search space, making small random changes to the solution. These changes may improve the solution or make it worse.

The algorithm uses a temperature parameter to control the probability of accepting a worse solution. At high temperatures, there is a high probability of accepting a worse solution, allowing the algorithm to escape local maxima. As the temperature decreases, the probability of accepting a worse solution decreases, leading to the convergence of the algorithm to a near-optimal solution.

Simulated annealing is particularly useful for solving problems where finding the optimal solution is difficult or time-consuming. It has been successfully applied to various problems such as the traveling salesman problem, the job shop scheduling problem, and the graph coloring problem.

In conclusion, simulated annealing is a powerful tool for problem-solving agents in artificial intelligence. It allows them to navigate complex search spaces and find near-optimal solutions. By using a probabilistic approach and gradually reducing the temperature, it can efficiently explore the solution space and overcome local maxima. This makes it an essential technique for solving challenging problems in artificial intelligence.

Genetic Algorithms

Genetic Algorithms (GAs) are a type of Artificial Intelligence (AI) technique used for problem-solving. They are commonly used in various fields, including optimization, machine learning, and data mining. GAs are inspired by the process of natural selection and evolution.

In a GA, a population of candidate solutions evolves over time to find the optimal solution to a given problem. Each candidate solution, also known as an individual, is represented as a set of chromosomes, which are strings of binary digits. The fitness of each individual is evaluated based on how well it solves the problem.

During the evolution process, individuals with higher fitness have a higher chance of being selected for reproduction. Genetic operators such as crossover and mutation are applied to the selected individuals to create new offspring. The new offspring then replace the less fit individuals in the population.

This process of selection, reproduction, and replacement continues over multiple generations until the population converges to a near-optimal solution. The convergence speed and quality of the solution depend on the parameters and characteristics of the GA, such as the selection method, crossover rate, mutation rate, and population size.

The use of GAs allows for the exploration of a large search space and can often find good solutions to complex problems. However, GAs do not guarantee finding the optimal solution, but rather give a good approximation.

TutorialsPoint is a great resource for learning about Artificial Intelligence, including Genetic Algorithms. They provide detailed tutorials and examples that help beginners understand the concepts and implementation of GAs. By following their tutorials, you can learn how to apply Genetic Algorithms to solve various problems in the field of AI.

Constraint Satisfaction Problems (CSPs)

In the field of artificial intelligence, problem solving agents often encounter situations where they need to fulfill a set of constraints in order to find a solution. These problems are known as Constraint Satisfaction Problems (CSPs). CSPs involve finding values for a set of variables that satisfy a given set of constraints.

A CSP can be defined as a triple (X, D, C), where:

• X is a set of variables, each with a domain of possible values.
• D is a set of domains, where each domain contains the possible values for its corresponding variable.
• C is a set of constraints, which define the relationship between the variables and their possible values.

Solving CSPs

Solving CSPs involves finding assignments of values to variables that satisfy all the given constraints. This can be done using various techniques, such as:

• Backtracking : This is a systematic search algorithm that explores the possible assignments of values to variables, backtracking when a conflict is encountered.
• Constraint propagation : This technique involves using the constraints to eliminate values from the domains of variables, reducing the search space.
• Constraint satisfaction algorithms : These algorithms use heuristics to guide the search for solutions, making it more efficient.

CSPs are used to solve a wide range of problems, such as scheduling, planning, resource allocation, and configuration. They provide a powerful framework for representing and solving problems in the field of artificial intelligence.

CSP Formulation

In the context of problem-solving agents, a CSP (Constraint Satisfaction Problem) is a formulation used to represent and solve problems in artificial intelligence. CSPs are widely used in various domains, such as scheduling, planning, and optimization.

In a CSP, the problem is represented as a set of variables, each with a domain of possible values, and a set of constraints that specify the relationships between variables. The goal is to find an assignment of values to variables that satisfies all the constraints.

Variables and Domains

Variables represent the unknowns in the problem, and each variable has a domain that defines the possible values it can take. For example, in a scheduling problem, the variables could represent tasks, and the domain of each variable could be the possible time slots for that task.

Constraints

Constraints define the relationships between variables. They specify the conditions that must be satisfied by the variable assignments. For example, in a scheduling problem, a constraint could be that two tasks cannot be scheduled at the same time.

Constraints can have different types, such as binary constraints that involve two variables, or unary constraints that involve only one variable. They can also have different degrees, such as constraints that involve more than two variables.

To solve a CSP, the agent searches for a consistent assignment of values to variables that satisfies all the constraints. This can be done using various search algorithms, such as backtracking or constraint propagation.

Let’s consider an example of a CSP formulation for a scheduling problem. We have three tasks to schedule, A, B, and C, and each task can be scheduled at two possible time slots, 1 and 2.

The variables are A, B, and C, and their domains are {1, 2}. The constraints are:

The goal is to find an assignment of values to variables that satisfies all the constraints. In this case, the valid assignments are:

• A = 1, B = 2, C = 1
• A = 2, B = 1, C = 2

These assignments satisfy all the constraints, as there are no conflicts between the tasks scheduled at the same time.

In conclusion, CSP formulation is a powerful technique used in problem-solving agents to represent and solve problems in artificial intelligence. It provides a flexible and efficient way to model and reason about complex problems.

Backtracking Algorithm

In the field of artificial intelligence, problem-solving agents are designed to find solutions to complex problems. One popular approach is the use of backtracking algorithms.

Backtracking is a systematic way of finding solutions by exploring all possible paths and discarding those that do not lead to a solution. It is often used when the problem can be represented as a search tree, where each node represents a partial solution and the edges represent possible choices.

The Backtracking Process

The backtracking algorithm starts by examining the first choice and moves forward along the path until a dead end is reached. At this point, it backtracks to the previous choice and explores the next option. This process continues until a solution is found or all possibilities have been exhausted.

During the backtracking process, the algorithm uses pruning techniques to optimize the search. Pruning involves eliminating portions of the search tree that are known to lead to dead ends, reducing the number of nodes that need to be explored.

Applications of Backtracking

The backtracking algorithm can be applied to a wide range of problem-solving tasks in artificial intelligence. Some common applications include:

• Constraint satisfaction problems: Backtracking can be used to find solutions that satisfy a set of constraints. For example, in a Sudoku puzzle, backtracking can be used to fill in the empty cells while ensuring that no number is repeated in a row, column, or block.
• Graph problems: Backtracking can be used to find paths in a graph that satisfy certain conditions. For example, in a maze, backtracking can be used to find a path from the starting point to the goal.
• Combinatorial optimization: Backtracking can be used to find the optimal solution among a set of possibilities. For example, in the traveling salesman problem, backtracking can be used to find the shortest possible route that visits all cities.

In summary, backtracking algorithms are a powerful tool for solving complex problems in artificial intelligence. They allow problem-solving agents to systematically explore all possible solutions and find the best one.

Forward Checking Algorithm

Forward Checking is an algorithm used in artificial intelligence to solve problems by systematically exploring the search space. It is particularly useful in constraint satisfaction problems where a set of constraints must be satisfied.

In the context of problem solving agents, the Forward Checking algorithm is used to efficiently eliminate values from the domains of variables during the search process. It works by propagating information about constraints from assigned variables to unassigned variables, reducing their domains and increasing the efficiency of the search.

The Forward Checking algorithm can be summarized in the following steps:

Step 1: Initialization

Initialize the domain of each variable with all possible values.

Step 2: Assign a Value

Select an unassigned variable and assign a value from its domain.

Step 3: Forward Checking

Update the domains of other unassigned variables based on the assigned value and the constraints.

In the table above, the domains of the variables are updated after assigning a value to one of the variables.

The Forward Checking algorithm continues by selecting the next unassigned variable with the smallest domain and repeating steps 2 and 3 until a solution is found or all variables have been assigned.

By efficiently propagating constraints, the Forward Checking algorithm can greatly reduce the search space and improve the efficiency of problem solving agents in artificial intelligence.

Constraint Propagation

Constraint propagation is one of the key techniques used by problem-solving agents in artificial intelligence. It is the process of using constraints to narrow down the search space and reduce the number of possible solutions.

In the context of artificial intelligence, a constraint represents a restriction or limitation on the values that certain variables can take. These constraints can be used to model real-world constraints or dependencies between variables. For example, in a scheduling problem, the constraint might state that two tasks cannot be scheduled at the same time.

Constraint propagation works by iteratively applying constraints to the problem domain and updating the values of variables based on the constraints. The goal is to eliminate values that are not consistent with the constraints, thus reducing the search space and making it easier to find a solution.

Types of Constraint Propagation

There are different types of constraint propagation techniques, including:

• Local Constraint Propagation: This technique applies constraints at a local level, focusing on individual variables or groups of variables. It updates the value of a variable based on the constraints without considering the global context of the problem.
• Global Constraint Propagation: This technique considers the global context of the problem and applies constraints across all variables simultaneously. It updates the values of variables based on the constraints and the implications of those constraints on other variables.

Constraint propagation can be a powerful technique for problem-solving agents as it allows them to prune the search space and focus on more promising solutions. It can help reduce the time and computational resources required to find a solution to a problem.

In conclusion, constraint propagation is a fundamental technique used by problem-solving agents in artificial intelligence. It leverages constraints to reduce the search space and find consistent values for variables. By applying constraints at a local or global level, agents can effectively narrow down the possibilities and improve the efficiency of their problem-solving process.

Local Search in CSPs

Local search is a common approach used in solving constraint satisfaction problems (CSPs) in artificial intelligence. CSPs involve finding solutions to a set of variables subject to a set of constraints. Local search algorithms focus on improving a current solution by making small modifications to it.

In the context of CSPs, local search algorithms explore the solution space by starting with an initial assignment of values to variables. They then iteratively improve this assignment by considering different neighborhoods of the current solution.

Local search algorithms aim to find the best possible assignment of values that satisfies all constraints in the problem. However, they do not guarantee finding the global optimal solution. Instead, they focus on finding a solution that is acceptable or meets certain criteria within a given time limit.

Tutorialspoint offers various tutorials and resources on local search algorithms in artificial intelligence. These tutorials provide in-depth explanations and implementations of different local search algorithms, such as hill climbing, simulated annealing, and genetic algorithms.

By learning and understanding local search algorithms, you can apply them to solve a wide range of problem-solving tasks in artificial intelligence. Whether it’s optimizing a complex scheduling problem or finding the best configuration for a system, local search algorithms provide practical solutions to real-world problems.

Tabu Search

Tabu Search is an artificial intelligence technique used for solving complex problem instances. It is a metaheuristic method that efficiently explores a problem space by keeping track of previously visited states, known as the tabu list, to avoid revisiting them. This allows the search algorithm to overcome local optima and find better solutions.

The main idea behind Tabu Search is to use memory to guide the search process. It maintains a short-term memory of recent moves and a long-term memory of the best solutions found so far. This allows the algorithm to make informed decisions and avoid getting stuck in suboptimal regions of the problem space.

During the search, Tabu Search uses different strategies to explore the problem space, such as generating and evaluating neighboring solutions, choosing the best move, and updating the tabu list. The tabu list contains forbidden moves that are temporarily avoided to prevent the search algorithm from going backward or cycling through the same solutions.

Tabu Search is particularly effective in solving optimization problems, combinatorial problems, and scheduling problems. It has been successfully applied in various domains, including operations research, computer science, engineering, and economics.

In conclusion, Tabu Search is a powerful technique used by problem-solving agents in artificial intelligence to tackle complex problem instances. It leverages memory to guide the search process and avoid revisiting previously explored states. By doing so, it is able to overcome local optima and find better solutions.+

Min-Conflict Algorithm

The Min-Conflict algorithm is a popular approach in the field of problem-solving intelligence. It is commonly used to solve constraint satisfaction problems.

This algorithm comes in handy when we need to find a solution that satisfies a set of constraints. It is especially useful when we encounter a problem where we have partial information or conflicting information.

The Min-Conflict algorithm works by iteratively adjusting the current solution to the problem until a feasible solution is found. It starts with an initial solution and then repeatedly selects a variable with a conflict and changes its value to minimize the number of conflicts. This process continues until either a solution with no conflicts is found or a predefined number of iterations is reached.

One of the advantages of the Min-Conflict algorithm is its ability to quickly find solutions to complex problems. It can handle large domains and a high number of constraints efficiently, making it a favored technique in artificial intelligence.

Implementing the Min-Conflict Algorithm

To implement the Min-Conflict algorithm, we need to follow these steps:

• Initialize the problem with an initial solution.
• While a solution with no conflicts is not found and the maximum number of iterations is not reached, repeat the following steps:
• Select a variable with conflicts.
• Choose a value for that variable that minimizes the number of conflicts.
• Update the assignments based on the new value.
• If a solution is found within the iterations limit, return it. Otherwise, return that no solution exists.

The Min-Conflict algorithm is a powerful approach for solving constraint satisfaction problems. Its iterative nature and ability to handle conflicts efficiently make it a preferred technique in artificial intelligence. By following a few simple steps, we can implement this algorithm to solve a wide range of complex problems.

Questions and answers

What are problem-solving agents in artificial intelligence.

Problem-solving agents in artificial intelligence are programs that are designed to find solutions to specific problems by searching through a set of possible actions and states.

How do problem-solving agents work?

Problem-solving agents work by starting with an initial state and applying a series of actions to reach a goal state. They use different search algorithms, such as depth-first search or breadth-first search, to explore the problem space and find the most optimal solution.

What are some examples of problem-solving agents?

Examples of problem-solving agents include route-planning systems, chess-playing programs, and automated theorem provers. These agents are designed to solve specific problems by searching for the best possible solution.

What are the advantages of problem-solving agents in artificial intelligence?

Problem-solving agents in artificial intelligence have several advantages. They can solve complex problems that would be difficult for humans to solve manually. They can also work quickly and efficiently, and they can explore a large number of possible solutions to find the best one.

What are some limitations of problem-solving agents in artificial intelligence?

Although problem-solving agents are powerful tools in artificial intelligence, they have some limitations. They require a well-defined problem and goal state, and they may not always find the optimal solution. Additionally, the search space can be very large, which can make finding a solution time-consuming or even infeasible.

Related posts:

About the author

Cohere: Bridging Language and AI for a Smarter Future

Microsoft office 365 ai. microsoft copilot, understanding the distinctions between artificial intelligence and human intelligence, exploring the impact of artificial intelligence on discrimination in insurance pricing and underwriting.

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

A novel and effective method for solving the router nodes placement in wireless mesh networks using reinforcement learning

Contributed equally to this work with: Le Huu Binh, Thuy-Van T. Duong

Roles Investigation, Methodology, Software, Writing – original draft, Writing – review & editing

Affiliation Faculty of Information Technology, University of Sciences, Hue University, Hue City, Vietnam

Roles Conceptualization, Investigation, Methodology, Writing – review & editing

* E-mail: [email protected]

Affiliation Faculty of Information Technology, Ton Duc Thang University, Ho Chi Minh City, Viet Nam

• Le Huu Binh, 
• Thuy-Van T. Duong

• Published: April 10, 2024
• https://doi.org/10.1371/journal.pone.0301073
• Peer Review
• Reader Comments

Router nodes placement (RNP) is an important issue in the design and implementation of wireless mesh networks (WMN). This is known as an P-hard problem, which cannot be solved using conventional algorithms. Consequently, approximate optimization strategies are commonly used to solve this problem. With heavy node density and wide-area WMNs, solving the RNP problem using approximation algorithms often faces many difficulties, therefore, a more effective solution is necessary. This motivated us to conduct this work. We propose a new method for solving the RNP problem using reinforcement learning (RL). The RNP problem is modeled as an RL model with environment, agent, action, and reward are equivalent to the network system, routers, coordinate adjustment, and connectivity of the RNP problem, respectively. To the best of our knowledge, this is the first study that applies RL to solve the RNP problem. The experimental results showed that the proposed method increased the network connectivity by up to 22.73% compared to the most recent methods.

Citation: Binh LH, Duong T-VT (2024) A novel and effective method for solving the router nodes placement in wireless mesh networks using reinforcement learning. PLoS ONE 19(4): e0301073. https://doi.org/10.1371/journal.pone.0301073

Editor: Mohammed Balfaqih, University of Jeddah, SAUDI ARABIA

Received: August 9, 2023; Accepted: March 9, 2024; Published: April 10, 2024

Copyright: © 2024 Binh, Duong. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting information files.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Wireless communication is growing and being widely applied in many fields. In the local area network of agencies, businesses, schools, and so on, wireless mesh networks (WMN) [ 1 , 2 ] are the best choice today because of their significant advantages compared to wireless networks using traditional access points. The most notable benefit of the WMN is that it reduces congestion owing to its ability to balance the loads. In addition, the installation of a WMN is very convenient because there is no need to construct wired connections from the gateway to all routers. Fig 1 illustrates an example of a WMN consisting of six mesh routers (represented by r 1 to r 6 ) and eleven mesh clients (represented by c 1 to c 11 ). In addition, at least one the router of the Internet service provider serves as a gateway for clients to access the Internet. If two mesh routers are within range of each other, a wireless link is established between them. A mesh topology consists of of all the mesh routers and wireless links. For a WMN to deliver Internet services, several mesh routers must be connected to the gateway router via wireless or cable links. As shown in Fig 1 , the mesh routers r 1 and r 2 are connected to the gateway router (GPON or FTTh router) via wireless links. Mesh clients are terminal devices that are users of network services. When a mesh client enters the network region, it can be covered by one or more mesh routers; the mesh client connects to the nearest mesh router to access network services.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0301073.g001

With the rapid development of wireless and mobile communication technologies, network services are becoming more diverse and rich, especially those on fifth-generation (5G) and sixth-generation (6G) wireless network platforms. To effectively provide these services, WMNs must be designed and installed in the most efficient manner possible, allowing network resources to be fully utilized. This is the motivation for researchers to focus on WMN. Some of the most prevalent subjects that have been implemented include network topology control [ 3 – 7 ], router node placement (RNP) [ 8 – 24 ], optimum routing protocols [ 25 – 29 ], and access point allocation [ 30 – 33 ], with the RNP challenge being the most fascinating. Because the RNP problem is known to be NP-hard, it cannot be solved using conventional algorithms. Recently, approximate optimization methods have become useful for solving this problem [ 8 – 12 ]. The authors of [ 8 ] have used the coyote optimization algorithm (COA) to solve the RNP problem. Their proposed method optimizes both network connectivity and user coverage, which are two critical performance criteria. Using MATLAB simulations, the authors demonstrated that the COA algorithm outperformed other well-known optimization algorithms. In [ 10 ], the authors suggested an optimal method called the Chemical Reaction Optimization (CRO) algorithm to solve this problem. The CRO algorithm was inspired by how molecules interact to achieve a low, stable energy state in chemical reactions. In terms of client coverage and network connection, the simulation findings reveal that their suggested approach outperforms the Genetic approach (GA) and Simulated Annealing (SA). Another study employed a genetic algorithm and simulated annealing to discover a low-cost WMN configuration while satisfying restrictions and identifying the number of gateways needed [ 34 ]. Experiments showed that the evolutionary algorithm and simulated annealing were successful in lowering WMN network expenses while maintaining QoS. The new models significantly outperformed the conventional solutions. QoS was also considered in the RNP problem in [ 23 ]. The authors described a unique particle swarm optimization method for improving network connectivity and client coverage. The QoS restrictions for this study are the delay, relay load, and Internet gateway capacity. In [ 35 ], the authors suggested an improved version of the Moth Flame Optimization (MFO) algorithm, namely, Enhanced Chaotic Lévy Opposition-based MFO (ECLO-MFO), for solving the RNP problem. To improve the optimization performance of MFO, the proposed method integrates three strategies: the chaotic map concept, Lévy flying strategy, and Opposition-Based Learning (OBL) technique. The simulation results showed that the proposed algorithm was more efficient than the method of applying popular optimization algorithms.

Based on the results of published works, we find that the method of using approximate optimal algorithms provide good solutions. However, because randomness is used in several steps of the algorithm, the results often differ for different executions. For accurate results, each script must be executed multiple times, and then the average of all executions is obtained. For example, the authors of [ 8 , 11 ] executed each simulation scenario 50 times. Furthermore, with heavy node density and wide-area WMNs, solving the RNP problem with approximation algorithms often presents many difficulties, necessitating a more effective solution. In this paper, we propose a new and effective algorithm to solve this problem. The main contributions of this study are summarized as follows:

• (i) We proposed a novel and effective method for solving the RNP problem using RL. The RNP problem is modeled as an RL model, with the environment, agent, action, and reward representing the network system, routers, coordinate adjustment, and connectivity respectively, of the RNP problem. To the best of our knowledge, this is the first study to apply reinforcement learning to the RNP problem.
• (ii) We compared and evaluated the performance of the RNP problem solving method using the heuristic algorithms and the RL method.

The remainder of this paper is organized as follows. The next section describes the formulation of the RNP problem in the WMN. The following sections present our proposed solution and experimental results. Finally, concluding remarks and promising future studies are presented in the last section.

RNP problem

In this section, we formulate the RNP problem in a WMN. First, graph theory was used to describe the WMN. We then define some metrics to use for the objective function of the RNP problem, similar to [ 11 ]. Finally, the RNP problem was formulated as a nonlinear programming problem. For convenience, we define the mathematical symbols shown in Table 1 .

https://doi.org/10.1371/journal.pone.0301073.t001

Mathematical model of a WMN using graph theory

Consider a WMN comprising m mesh routers, n mesh clients, and k gateway routers. Mathematically, this WMN can be represented as an undirected graph, denoted by G = ( V , E ), where V and E are the vertex and edge sets, respectively. V is equivalent to the set of all nodes in the WMN and is determined by V = R ∪ C ∪ W , where R , C and W are the sets of mesh routers, mesh clients, and gateway routers, respectively. E is equivalent to the set of all wireless links in the WMN and consists of three types: links between mesh routers, links between mesh client and mesh router, and links between gateway and mesh router.

RNP problem formulation

In this section, we formulate the RNP problem using some concepts and metrics from [ 11 ], including the connected router, connected client, connected router ratio, and connected client ratio.

Connected router.

The mesh router r i is a connected router if and only if at least one path exists between it and the gateway router. If we return to the WMN example in Fig 1 , we can see that mesh routers r 1 , r 2 , r 3 , r 4 and r 6 are the connected routers but r 5 is not because no path exists from this mesh router to the gateway router.

Connected router ratio (CRR).

Connected client.

Connected client ratio (CCR).

Formulate the RNP into a nonlinear programming problem.

The RNP problem in the WMN is stated as follows: Consider a case where it is necessary to design and install a WMN with the following assumptions:

• The network system is located in an area of W × H meters.

• The number of mesh routers was m , and the coverage radius of each mesh router was d r .

RL-based mesh router nodes placement

Fundamentals of rl.

https://doi.org/10.1371/journal.pone.0301073.g002

RL has been successfully applied to control protocols in wireless networks, typically routing in WMN [ 25 , 27 , 29 ], topology control in wireless sensor networks [ 37 ], improving the performance of energy-harvesting wireless body area networks [ 38 , 39 ]. In this paper, we apply RL to solve the RNP in WMN. Details of this new proposal are presented in the following sections.

Solving the RNP in WMN using RL

The RL has recently been successfully employed to solve technical challenges in wireless communication such as routing [ 27 , 36 ], topology management [ 37 ], and resource allocation. In this study, we use RL to solve the RNP problem. To the best of our knowledge, this is the first study to use RL to address the RNP problem. To do this, the RNP problem must be modeled as a reinforcement learning model with five characteristic factors: agent, environment, state, action, and reward.

An agent is a mesh router that regularly adjusts its coordinates to obtain an optimal topology.

Environment.

In a RL model, the environment is everything that exists around the agent, and it is where the agent acts and interacts. The environment for the RNP problem using RL is the network system, which includes a set of mesh routers, clients, gateway routers, and network area.

Each state is determined by a triple { P c , P r , P w }, where P c , P r and P w are the sets of coordinates for the mesh clients, mesh router, and gateway routers, respectively. The sets are listed in Table 1 .

Action is the way in which the agent interacts with the environment to change its state. For the RNP problem using reinforcement learning, the agents are the mesh routers. Each action was defined by a mesh router that adjusted its coordinates. The set of actions at a specific state s t for each mesh router r i is defined as A t = { mn1s, me1s, ms1s, mw1s, mn2s, me2s, ms2s, mw2s }, where the actions are described in Table 2 , step is a given distance.

https://doi.org/10.1371/journal.pone.0301073.t002

RL algorithm for solving RNP problem.

Algorithm 1 The pseudo-code of the reinforcement learning algorithm for solving RNP problem

• Network area ( W × H );

• The set of mesh routers ( R = { r i | i = 1.. m }), and the coverage radius of each mesh router ( d r );

2: while ( learn ≤ numLearn ) do

3:  Randomly choose mesh router r i ∈ R ;

4:   for ( each action a j ∈ A ) do

5:   Update Q ( r i , s t , a t , j ) using ( 11 );

6:   end for

7:  Choose the action a t , k ∈ A t using policy derived from Q-values (e.g., ε -greedy ) according to ( 12 );

8:  Take action a t , k , observe reward R ( r i , s t , a t , k ) and next state s t +1 ;

9:  Update next state ( s t +1 ) to current state ( s t );

10:   learn ← learn + 1;

11: end while

12: P r ← P r in state s t ;

Analyze the computational complexity.

The computational complexity of Algorithm 1 depends mainly on the iteration in Step (2), the number of possible actions in Step (4), and the algorithm for updating the Q value in Step (5). Q ( r i , s t , a t , j ) is updated using Eq. ( 11 ), where the greatest complexity is the calculation of RW ( r i , s t , a t ) according to ( 10 ). RW ( r i , s t , a t ) contains two metrics, CRR and CCR , which are defined by ( 1 ) and ( 5 ), respectively. To determine CRR , we employed a breadth-first search algorithm on a network of m vertices, which is the number of mesh routers. Therefore, the computational complexity was O ( m 2 ). The CCR is calculated using two nested loops of sizes m and n , where n is the number of mesh clients. Therefore, the complexity was O ( m × n ). Because n is always greater than m in a WMN, the computational complexity of RW ( r i , s t , a t ) is O ( m × n ). Consequently, the computational complexity of Algorithm 1 is O ( I × | A | × m × n ), where I is the number of iterations and | A | is the number of possible actions.

The computational complexity of Algorithm 1 is greater than that of the algorithms solving the RNP problem using GA [ 40 ], PSO [ 24 ], and WOA [ 41 ], which we compare in the following section. However, because its computing complexity is a polynomial function, it can be implemented in practice. Furthermore, because the algorithms for solving the RNP problem are run offline, the polynomial complexity is acceptable.

Simulation results and discussion

Simulation scenarios.

The performance of the proposed method was evaluated through a simulation using Python. Our proposed method is compared with the most recent methods that use approximate optimization algorithms to address the RNP problem, including GA [ 40 ], PSO [ 24 ], WOA [ 41 ], and MVO [ 11 ]. All experiments were run on a 3.6 GHz Core i7 CPU computer. The surveyed network instances (NI) are presented in Table 3 . NI-1 and NI-2 were used to investigate the effect of the number of mesh routers on the network performance, with the number of mesh routers ranging from 20 to 45 covering 150 mesh clients (NI-1) and 350 mesh clients (NI-2). NI-3 and NI-4 ware used to study the effect of client density, varying from 100 to 400. In NI-5 and N-6, the effect of the coverage radius of each mesh router was thoroughly examined. The final two NIs were used to investigate the influence of the network area. The parameters of the simulation scenarios and algorithms are presented in Table 4 , where th parameters of the GA, PSO, WOA, and MVO are set as in [ 11 ].

https://doi.org/10.1371/journal.pone.0301073.t003

https://doi.org/10.1371/journal.pone.0301073.t004

Simulation results

Topology evaluation..

First, we evaluate the topology obtained when solving the RNP problem using the GA, PSO, WOA, MVO, and our proposed method, which employs reinforcement learning. The results obtained in Fig 3 clearly show topological differences between the methods. These findings were obtained using NI-2 with 30 mesh routers covering 350 mesh clients in an area of 2000 × 2000 [ m 2 ] and a coverage radius of 200 [ m ] for each mesh router. We can observe that the method using RL provides the most optimal topology compared with the methods using approximate optimization algorithms, GA, WOA, PSO, and MVO. Specifically, for the method using reinforcement learning, there are 334 mesh clients covered by at least one mesh router, corresponding to a rate of 95.43%. These values were 292 (83.43%), 309 (88.29%), 313 (89.43%), and 313 (88.86%) for the WOA, GA, PSO, and MVO algorithms, respectively. In addition, the topology of the reinforcement learning method has a wider coverage area than the other methods, which can increase the percentage of clients covered in the case of denser clients.

(a) WOA, (b) GA, (c) PSO, (d) MVO, and (e) reinforcement learning.

https://doi.org/10.1371/journal.pone.0301073.g003

Impact of mesh router density.

The results obtained in Fig 4 clearly show the difference in network connectivity between the proposed method and the method using approximate optimization algorithms. These findings were obtained using NI-1, in which the number of mesh routers varieed from 20 to 45, covering 150 mesh clients in an area of 2000 × 2000 [ m 2 ] and a coverage radius of 200 [ m ] for each mesh router. We can observe that the NC increases proportionally with the number of mesh routers for all methods. This is evident because as the number of mesh routers increases, the coverage area expands, increasing the probability of mesh clients being covered. Comparing the methods of solving RNP problems, the method using RL (legend namely RL-based RNP) gives the highest NC. For example, considering the case of 35 mesh routers, The NC values of the methods using the WOA, PSO, GA, MVO, and RL are 85.64, 87.42, 90.67, 93.42, and 95.68%, respectively. Thus, compared with the method using algorithms WOA, PSO, GA, and MVO, the proposed method improved NC by 10.03, 8.25, 5.01%, and 2.25%, respectively. This is a significant result in improving WMN performance.

https://doi.org/10.1371/journal.pone.0301073.g004

The results obtained were quite similar for the implementation on NI-2, as shown in Fig 5 . The assumptions of this simulation scenario are the same as those in NI-1, except that the number of mesh clients increases to 350. We can see that the proposed method is highly effective in terms of NC. We can observe that the proposed solution provides high efficiency in terms of NC for most values of the number of mesh routers. The NC of the method using RL increases by an average of 4 to 20% compared with the cases where approximate optimization algorithms are used. As is the case with 35 mesh routers, the NC of the RL is 98.71%. These values of the WOA, PSO, GA, and MVO algorithms were 81.66%, 86.89%, 88.59%, and 94.44% respectively. Thus, the method using RL improved the NC from 4.26% to 17.04%.

https://doi.org/10.1371/journal.pone.0301073.g005

Based on the findings in Figs 4 and 5 , we can conclude that changing the number of mesh routers affects on network performance in terms of NC. The larger the number of routers, the higher the NC for all investigated RNP problem solving methods. In particular, the method based on RL is the most efficient.

Impact of mesh client density.

In this section, we investigate the effect of client density on network performance. In a WMN, the denser the clients, the greater is the number of connection requests to the routers. As a result, network performance was affected. This is more evident in Fig 6 , where we plot NC as a function of the number of mesh clients. These results are obtained by executing NI-3, where the number of mesh routers is 30, covering 150 to 300 mesh clients. We can easily observe that the method using RL always yields the highest NC regardless of whether the client density is sparse or dense. The NC value of this method from 90.43% to 95.79%. Meanwhile, the NC values for the cases of algorithm WOA, PSO, GA, and MVO are fom 74.59% to 84.08%, from 77.00% to 85.63%, from 82.32% to 91.46%, and from 88.27% to 90.83%, respectively. When 45 mesh routers were used (NI-4), the NC value increased for all methods. This is clearly shown in Fig 7 , where we represent NC versus the number of mesh clients. Comparing the methods, we find that the method using RL outperforms the method using approximate optimal algorithms in terms of NC.

https://doi.org/10.1371/journal.pone.0301073.g006

https://doi.org/10.1371/journal.pone.0301073.g007

Impart of the coverage radius of mesh routers.

The coverage radius of the mesh routers is another technological parameter that has a considerable impact on the WMN performance. In this section, we investigate the effect of this technological parameter on the NC metric. The results obtained in Fig 8 clearly show the change in NC with respect to the coverage radius of the mesh routers. These results were implemented using NI-5, which has 30 mesh routers and 150 mesh clients. The coverage radius of each router ranged from 150 to 300 [ m ]. The plots in Fig 8 indicate that the NC increases proportionally to the coverage radius of the mesh routers. This is because expanding the coverage radius increases the likelihood that clients will be covered. As a result, NC increases. In particular, the method using RL yielded the highest NC, reaching close to 100% when the coverage radius was 250 [m] or more. The results are also similar for NI-6, as shown in Fig 9 . The NC value of this NI is greater than that of the NI-5 because this uses more mesh routers. As in the previous scenarios, the method using RL always yields the highest NC.

https://doi.org/10.1371/journal.pone.0301073.g008

https://doi.org/10.1371/journal.pone.0301073.g009

Impact of network area.

In the last section, we investigate the effect of network area on the efficiency of RNP problem solving methods. Figs 10 and 11 show the results obtained by executing NI-7 and NI-8, respectively. In these NIs, the network area varies from 2000 × 2000 [ m 2 ] to 3000 × 3000 [ m 2 ]. The NC value decreased according to the network area for all the algorithms. This is because, for a given number of mesh routers, the larger the network area, the lower the percentage of area covered, leading to a decrease in the NC value. However, the NC value of the method using RL is always the largest.

https://doi.org/10.1371/journal.pone.0301073.g010

https://doi.org/10.1371/journal.pone.0301073.g011

Based on the above findings, we can conclude that the proposed method, which uses reinforcement learning to solve the RNP problem, is more efficient than a method that uses approximate optimal algorithms. This is a crucial result in the design and implementation of a WMN, which helps find an optimal network topology to exploit network resources more efficiently.

The placement of router nodes in wireless mesh networks is a significant problem that has recently attracted the interest of several research groups. This problem is recognized as NP-hard, and cannot be resolved using conventional algorithms. In this study, we proposed a new and effective method for solving this problem using RL. The process of finding the optimal coordinates for placing mesh routers is modeled as an RL with the main components being environment, agent, action, and reward, which are equivalent to the network system, routers, coordinate adjustment, and network connectivity of the RNP problem, respectively. Simulation results show that our proposed method outperforms the most recent methods in terms of coverage and network connectivity.

In future work, we will continue to develop this method by considering additional constraints on the quality of transmission and load balancing to improve network performance. In addition, the deep reinforcement learning method can also be applied to static and dynamic RNP problems to further improve the performance of the WMN.

Supporting information

S1 dataset..

https://doi.org/10.1371/journal.pone.0301073.s001

• 1. Zhang Y, Luo J, Hu H. Wireless Mesh Networking—Architectures, Protocols and Standards. Taylor & Francis Group, LLC; 2007.
• 2. Akyildiz IF, Xudong Wang. Wireless Mesh Networks. John Wiley & Sons Ltd; 2009.
• View Article
• Google Scholar
• 4. Aron FO, Olwal TO, Kurien A, Odhiambo MO. Energy Efficient Topology Control Algorithm for Wireless Mesh Networks. In: 2008 International Wireless Communications and Mobile Computing Conference; 2008. p. 135–140.
• 6. Yang L, Quan L. A Topology Control Algorithm Using Power Control for Wireless Mesh Network. In: 2011 Third International Conference on Multimedia Information Networking and Security; 2011. p. 141–145.
• PubMed/NCBI
• 15. Bello OM, Taiwe KD. Mesh Node Placement in Wireless Mesh Network Based on Multiobjective Evolutionary Metaheuristic. In: Proceedings of the International Conference on Internet of Things and Cloud Computing. ICC’16. New York, NY, USA: Association for Computing Machinery; 2016.Available from: https://doi.org/10.1145/2896387.2896444 .
• 18. Hamdi M, Mhiri S. Dynamic mesh router placement for connectivity maximization in wireless mesh networks; 2015. p. 1–6.
• 20. Sayad L. Optimal placement of mesh routers in a wireless mesh network with mobile mesh clients using simulated annealing. In: 2017 5th International Symposium on Computational and Business Intelligence (ISCBI); 2017. p. 45–49.
• 21. Rezaei M, Sarram M, Derhami V, Sarvestani H. Novel Placement Mesh Router Approach for Wireless Mesh Network. 2012;.
• 22. Seetha S, Anand John Francis S, Grace Mary Kanaga E. Optimal Placement Techniques of Mesh Router Nodes in Wireless Mesh Networks. In: Haldorai A, Ramu A, Mohanram S, Chen MY, editors. 2nd EAI International Conference on Big Data Innovation for Sustainable Cognitive Computing. Cham: Springer International Publishing; 2021. p. 217–226.
• 23. Lin CC, Chen TH, Jhong SY. Wireless mesh router placement with constraints of gateway positions and QoS. In: 2015 11th International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness (QSHINE); 2015. p. 72–74.
• 35. Mekhmoukh Taleb S, Meraihi Y, Mirjalili S, Acheli D, Ramdane-Cherif A, Benmessaoud Gabis A. Mesh Router Nodes Placement for Wireless Mesh Networks Based on an Enhanced Moth–Flame Optimization Algorithm. Mobile Networks and Applications. 2023. https://doi.org/10.1007/s11036-022-02059-6