Mob mechanics is a broad term that encompasses the way mobs spawn, move, and interact with players. Different mobs have different behaviors and movement algorithms, so for specific information about a given mob, you should go to that mob's individual page. However, there are aspects of mob mechanics that allow for some generalization.
Note that most of the following information is based on players' observations. Overall, the developers don't tend to weigh in much on the specifics of mob mechanics. In addition, the terms here do not necessarily reflect any official or accepted nomenclature. For the most part, this page focuses on the way mobs spawn and move, and is intended primarily for those looking to build better mob farms, as well as anyone interested in the more technical and esoteric aspects of mob mechanics.
Mob spawning is the overall term for the various ways that mobs end up in a world. The two main types of spawning are maw spawning and area spawning. Other types of spawning are listed below, as well.
Area Spawning: Area spawning refers to the phenomenon of mobs appearing randomly in certain areas, as opposed to spawning from maws or in any of the other ways listed below. For birds and butterflies, this area includes any sky area. For all other mobs, this area is any place below the surface that's not covered by earth, blocks, items, or crafted background blocks.
Other Types of Mob Spawning
Note that the following kinds of mob spawning don't play a role in overall mob mechanics, and are included in the interests of thoroughness. The mobs themselves, however, do generally follow the same movement algorithms as naturally-spawned mobs.
Mined Item Spawning: This type of spawning is when you mine something and a mob suddenly appears. You'll see this when you're mining earth, or when you mine certain containers, such as crates and industrial crates. In the case of earth-mining, the mob that spawns is invariably a terrapus. When mining industrial crates, however, the mob is a small automaton.
Egg Spawning: Egg spawning is when a terrapus egg breaks, releasing a terrapus. It's sort of like mined item spawning, except you don't have to actually mine the item -- you just have to get close enough to it.
Mob-Controlled Spawning: This type of spawning is when one mob spawns another mob, and only applies to brain lords and terrapus queens, both of which will spawn smaller versions of themselves when a player draws too near.
Spawner Spawning: This refers to the spawning that happens when a player sets up a spawner. Note that spawner-spawned mobs have some special characteristics with regard to spawning and despawning. For more information about such mobs, see the page on spawners.
Evoker Spawning: On brain worlds that still have evokers, every ten minutes, five brains will spawn near you.
Vengeful Spawning: If you kill too many adult brains in too short a time, adult (and/or, occasionally large) brains will begin to retaliate by spawning right on top of you. You may notice this type of spawning especially at brain-farming worlds.
Dev Spawning: This includes 1) Spawn of Laurence (i.e., when Mike is in a world, and people are being annoying, and suddenly 47 brain lords appear); 2) Fish Spawning (only happens when Lisa is in a world, and almost always happens when Lisa is in a world. The fishes. Lisa likes them!); and 3) Bomb Spawning (an extremely rare type of spawning that involves the sudden appearance of bombs when you're in the same world as Jason and you invade his personal space bubble).
Mob movement depends on whether the mob is the kind of mob that flies around, or the kind that stays on the ground. Flying mobs seem to follow a single basic movement algorithm, with additional mechanics if they're hostile, while ground-based mobs follow one of two algorithms depending on whether they're crawling mobs or walking mobs.
Crawling mobs move only along surfaces, including up and down vertical surfaces (walls). They can't crawl upside down along ceilings, however. They're also unable to "jump through" two blocks placed kitty-corner (diagonally, with only the corners touching). In general, all crawling mobs will keep walking in one direction until they reach an obstacle, at which point they turn around and head back in the other direction. Some crawling mobs, such as terrapi, will randomly pause, and then start moving again in the same direction. Others, like roaches, will always move without pausing.
In all cases, crawling mobs will skip over a one-block-wide chute in a flat floor. In addition, they'll often refuse to "fall" out of a one-block-wide chute in a flat ceiling. With the exception of crawling baby brains, hostile crawling mobs will shoot when they notice a player, but won't stop or change directions to pursue. Crawling baby brains will follow players and change directions to pursue. All crawling mobs are also affected by conveyors. Incidentally, there are no crawling mobs that are bigger than 1x1 block.
Walking mobs are those that move by walking along the ground. Unlike crawling mobs, however, which can climb walls, walking mobs can only walk along flat surfaces and make jumps of up to 2 blocks. If faced with an obstruction 3 or more blocks tall, a walking mob will simply turn around and start walking in the opposite direction. Wallking mobs other than crawling baby brains and friendly ghosts are not coded to follow the player. Like crawling mobs, hostile walking mobs will notice a player and shoot, but will not stop or change directions to pursue. All walking mobs are affected by conveyors.
Walking mobs are unique in being able to "jump through" certain types of corners. As a result, they can be more difficult to capture and contain than other mobs. Like crawling mobs, they will almost always skip over a one-block-wide chute in a flat surface. (The android is a notable exception to this rule.) Unlike crawling mobs, they'll drop down out of one-block-wide chutes in flat ceilings. Some walking mobs will pause randomly (e.g., rats), while others (e.g., worms) never pause.
All crawling and walking mobs, with the exception of androids, will skip over one-block-wide chutes in flat floors.
All crawling and walking mobs, with the exception of crawling baby brains and ghosts, will continue to walk or crawl in the direction they were going until they reach an obstacle. Ghosts and baby brains that are not near players will behave like any other walking mob.
All crawling and walking mobs, if falling, will move in the direction they're facing when they land.
All crawling and walking mobs, with the exception of androids, when falling next to a vertical wall, will opt to turn and move horizontally if given the chance. This is the only way to get a crawling or walking mob to change directions without using obstacles. If there are no horizontal passages, they will not randomly change directions in mid-fall (except for ghosts and crawling baby brains, which will change direction if the player does).
All crawling and walking mobs, regardless of size, are able to navigate vertical chutes and horizontal tunnels that are one block wide or tall. There's nothing more disturbing than watching a terrapus queen slide ominously down a one-block-wide chute.
There are a variety of flying mobs in the game, some of which are hostile and some of which are just annoying. They can be very large (like brain lords) or very small (like butterflies). Some flying mobs always spawn in groups, like crows and bats. In the case of non-hostile flying mobs, movement is somewhat random, although anyone who's ever stood on a block of wood in the sky knows that every crow in the vicinity wants to perch on that block of wood and push you off. Bats, meanwhile, will hang upside down from the underside of blocks, and seem coded to do absolutely everything in their power to interfere with dungeon-raiding. (And most dungeons automatically spawn bats inside the dungeon alongside guardian brains.) In general, however, non-hostile flying mobs go about more or less independently of player activities.
Hostile flying mobs, meanwhile, will notice, follow, and shoot at any player that comes into visual range. Brains, revenants, and flying automata will all notice you and flock to you with ill intent. In general, when hostile flying mobs can't see a player to follow, they seem to revert back to default flying mob behavior, hovering around sniffing at the boundaries of whatever space they're in, moving randomly back and forth and up and down, and then eventually moving upwards if they can. Most flying mobs, if funneled into a one-block-wide chute, will tend to go up rather than down. Most flying mobs will avoid immediately going up a one-block-wide chute in a flat ceiling or going down a one-block-wide chute in a flat floor. They will readily rise up chutes at the corners of rooms, however. All flying mobs are unaffected by conveyors.
Special Mob Movements
Android: Every world automatically has 6 persistent androids. On every newly-generated world, these androids exist in a random distribution at the very top of the world, where they wait in a dormant state until a player's sphere of influence activates them to "spawn." They are the only mob able to jump down one-block-wide chutes in flat surfaces.
Newton Android: Every public world automatically has a Newton Android at center spawn. This android doesn't walk or fly, but will fall if the spawn structure has a gap under where the android spawns. Newton Androids are part of the Quest system.
Brain: All brains except baby brains have the ability to temporarily shovel through any block or item that is placed on top of any part of them. Brain lords, in particular, will shovel if they see a one-block-wide or larger opening, or if they spawn partially covered in blocks of any kind.
Baby Brain: Baby brains can spawn as either crawling mobs or as flying mobs. Crawling baby brains are the only hostile walking or crawling mob that will change directions to pursue a player.
Worm: Worms are essentially walking mobs with the ability to shovel their way through earth and other shovel-able materials. When they reach an obstacle they can't shovel through, they'll turn around and go the other way. These mobs do not randomly pause.
Ghost: Ghosts are the only non-hostile mobs specifically designed to exhibit following behavior. This is so you can lead them to the expiator.
Tiny Flying Automaton: Small flying automata may spawn with no weapons and no ability to deal damage on contact, but they don't seem to realize this: they'll still follow you and bump into you as aggressively as any brain.
Fish: Fish are not yet fully released into gameplay (February 2016), but you can view them at the zoo at Deepworld HQ and observe their movements for yourself.
Other Mob-Related Mechanics
Currently (February 2016), most mobs have some degree of collision with players. This means they will impede or affect your movement. The only mobs with zero player collision are ghosts and revenants. However this mechanic is in flux, and word on the street is that eventually no mobs will have such dramatic collision. Instead, all mobs will slow you down somewhat, and hostile mobs will slow you down more.
In order to make it more difficult for players to exploit brain farms for resources and XP, the mechanic of vengeful brains results in retaliation if you kill too many adult brains in too short a time, whether by exploders or any other method. If you kill more than 8 adult brains in a relatively short period of time, adult and large brains will begin spawning as close to you as possible, regardless of whether there are viable spawning areas. Note that this nerf only applies to killing adult brains (since they are the only brain type amenable to farming).
If a mob usually drops an item that can be placed (such as giblets or scrap metal), when you kill that mob via an exploder or critical hit, it will drop its item on the ground rather than into your inventory. Items that can't be placed (like jerky or canisters) will go into your inventory as usual. For more information, see the exploders page.
Sphere of Influence
Every player is surrounded by an invisible circle that influences the spawning and/or activation of mobs. The outer edge of this circle seems to be somewhere between the size of a large protector and a mega protector. In other words, your presence will affect mobs within about a 30 to 60 block radius of you.
If you're the only person on an entire world, the only mob activity is going to be within your sphere of influence. Mobs on the other side of the world are not dropping from maws; they're only spawning wherever you go. As you move about the world, mobs that are no longer in your sphere of influence will despawn or, in the case of persistent mobs like androids and pets, become inactive. Usually, you must be in an area for a second or two before spawning begins. So if you drop down a chute at full speed, mobs won't spawn along the way. But once you pause, your sphere of influence goes into effect.
Your sphere of influence also seems to have a sort of reverse effect on area spawning. That is, mobs don't tend to spawn randomly right on top of you. If you've ever been alone in a hell world, standing at the expiator and wishing for ghosts, you've seen this effect. Those ghosts just never seem to materialize right in front of you. It's unclear what the range of this sphere of influence is, but it seems to be somewhere between 30 and 40 blocks away in every direction. (You can, however, override this mechanic with a diamond or onyx mass spawner.)
Mob Population Limits
For every person on a world, there will be up to 8 naturally-spawned mobs (via maws or area spawning). Well, technically, the game seems to allow for eight instances of spawning per player. If all eight of those spawning events were crows, for example, you'd actually end up with 24 mobs, since crows always spawn in threes. Like poison ivy leaves.
Mobs will attempt to spawn within a player's sphere of influence when that player enters a world. If there's no viable place to spawn (for example, if all maws near the world teleporter have been plugged and all open areas have been filled or otherwise rendered unsuitable for spawning), mobs will spawn in the sphere of influence of the closest player (assuming there is a viable spawning option within that sphere of influence). If there's no other player on the world, no mobs will spawn until the player goes somewhere on the world that puts their sphere of influence over a viable spawning location (like an open maw, open background area, or, if in a world with birds, open sky).
To illustrate, imagine you've plugged every maw and culvert in a world except one, and have also made all area spawning impossible (perhaps by putting background blocks over every open underground and sky area, by simply filling in every opening with earth, or by using a diamond or onyx mass spawner to turn area spawning off altogether). The world teleporter is on one side of the world, and the open maw is on the other side of the world. If you stand near the maw, 8 mobs will spawn. If you don't kill any of them, no more will spawn, no matter how long you stand there.
Now if you have a friend enter the world and just stand at the world teleporter, 8 more mobs will spawn at your maw, even though your friend's sphere of influence is nowhere near the maw. Even if there's a viable spawning area closer to your friend, as long as it's outside your friend's sphere of influence, the mobs will spawn in your sphere of influence and not at the closer viable spot. In other words, for a viable spot to spawn mobs, it must be within an active sphere of influence.
It's not known if there's an upper limit to how many mobs can be active on a world at any given time. If the max number of people on a world is 30, and the formula of 8 mobs per player holds beyond 2 players, then in theory the max number of naturally-spawned mobs that can exist on a world at the same time would be 240.
Note that mobs spawned in ways other than maw spawning and area spawning are exempt from this general formula. So, for example, if you put 100 terrapus eggs down, all 100 of them will spawn terrapi if you break them at the same time. If you build a house and release 100 pet cats into it, the world will still attempt to spawn 8 mobs for you. It's not known if the mobs that terrapus queens and brain lords spawn affect the 8-mobs-per-player rule.